Tutorial: Host a website for a small company on IBM Cloud: Real-time auditing by Kubernetes with Falco

Securing Kubernetes evolves from a fleeting worry to a first class prerequisite as Kubernetes becomes the platform of choice for both software creation and deployment. This guide covers auditing the operation of Kubernetes clusters in real time and developing a framework to log and manage audit events automatically.

API powered is Kubernetes. In several implementations, the API interacts with each user, administrator, and developer. When a user issues a deployment command created by kubectl, it hits the API and records the request via the audit mechanism of Kubernetes. There are also non-human API interactions to record and inspect, operators run with their own collection of keys explicitly against the API, and cloud vendor external systems touch the API. If out-of-compliance activity is observed, all of these interactions should be logged, audited, and processed with notifications sent out.

Pre-Requirements

1. Account with IBM Cloud. You must first have to create an account with IBM. if you didn't have click this link to signup https://cloud.ibm.com/registration?cm_sp=ibmdev-_-developer-tutorials-_-cloudreg.

2. IBM Cloud Infrastructure Node Kubernetes. Follow this link to enter https://cloud.ibm.com/kubernetes/catalog/cluster?cm_sp=ibmdev-_-developer-tutorials-_-cloudreg

Measures

Step 1 Configure auditing by Kubernetes

Step 2. Establish auditing at Kubernetes

Stage 3. Set up the redirection of events from Falco to LogDNAA

Stage 1. Configure auditing by Kubernetes

Lets go. Any time the Kubernetes API is accessed, Kubernetes can be configured to issue audit events. In order to quickly identify perpetrators of our malicious activity, you can process these incidents with Falco in real time.

1. Using the IBM Cloud Virtual Private Cloud (VPC) service to build a virtual machine (VM) (Generation 2).

2. Get the floating public IP address of your virtual machine's IP.

$ ibmcloud target -r us-east
Switched to region us-east

API endpoint:      https://cloud.ibm.com
Region:            us-east
User:              skrum@us.ibx.com
Account:           Cloud Open Sauce (xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) <-> xxxxxxx
Resource group:    No resource group targeted, use 'ibmcloud target -g RESOURCE_GROUP'
CF API endpoint:
Org:
Space:
$ ibmcloud is instances
Listing instances for generation 2 compute in all resource groups and region us-east under account Cloud Open Source as user skrum@us.ibm.com...
ID                                          Name              Status    Address        Floating IP     Profile    Image
                              VPC       Zone        Resource group
xxxx_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   nibz-falco-dev    running   10.241.128.4   52.xxx.xxx.xxx   bx2-8x32   ibm-ubuntu-18-04-1-minimal-amd64-2   nibz      us-east-3   Default

If you do not have a floating IP set, return to the console to configure one. With the CLI, it is possible to set one (see ibmcloud is instance-network-interface-floating-ip-add, floating-ip-reserve, and instance-network-interface-create).

1. Install python to allow Ansible to run

ssh ${remote_ip} 'sudo apt-get update; sudo apt-get -y install python'

2. Set up an inventory list for Ansible, replacing 192.168.0.10 with the server's IP or DNS name..

[cloud]
192.168.0.10 ansible_ssh_user=ubuntu

3. Initialize ansible-galaxy.

 ansible-galaxy init

1.. Get the ansible-sshd Ansible role.

ansible-galaxy install willshersystems.sshd

2. Get the ansible-falco Ansible role.

git clone https://github.com/juju4/ansible-falco

Create a playbook file to use the Ansible role. Note the hosts: field if you customized anything in the inventory file.

---
- hosts: cloud
 become: yes
 vars:
   falco_grpc_unix_enabled: true
   falco_webserver_enable: true
   #    falco_dev: true
   sshd:
     GSSAPIAuthentication: no
     ChallengeResponseAuthentication: no
     PasswordAuthentication: no
     PermitRootLogin: no
 roles:
   - role: willshersystems.sshd
   - role: ansible-falco

Run the Ansible playbook to install Falco.

ansible-playbook -i inventory falco-install.yaml

SSH to host to validate the configuration.

ps -ef | grep falco

tail -f /var/log/falco/falco.log

The notices that Falco is running should be seen, but nothing is unique to Kubernetes.

The 2nd stage. Establish auditing at Kubernetes

The Kubernetes daemons are managed by the IBM Cloud Kubernetes service for you. Configuring the Kubernetes audit involves configuring the command line arguments of the Kubernetes API server. IBM Cloud offers an audit webhook management system and where it points. In the product documentation, read all about it.

ibmcloud ks cluster master audit-webhook set --cluster <cluster_name> --remote-server http://<server_floating_ip>:8765/k8s_audit
ibmcloud ks cluster master audit-webhook get --cluster <cluster_name>

# refresh (takes a few seconds)
ibmcloud ks cluster master refresh --cluster <cluster_name>

Poke a hole in the firewall or security group for your instance for Kubernetes audit events.

ibmcloud is security-group-rule-add xxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx inbound tcp --port-min 8765 --port-max 8765 --output JSON
{
   "direction": "inbound",
   "id": "xxxx-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "ip_version": "ipv4",
   "port_max": 8765,
   "port_min": 8765,
   "protocol": "tcp",
   "remote": {
       "cidr_block": "0.0.0.0/0"
   }
}

That should be it. Now you can check out your logs!
Edit your rules by modifying the Kubernetes rules under /etc/falco.
View the logs with the following command:
```
tail -f /var/log/falco/falco.log
```

  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:20.110256128: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:22.139158016: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:23.199183104: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:25.244869888: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:26.293675008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:28.339374080: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:29.416363008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:31.453209088: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-controller-manager target=kube-scheduler/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=15s resp=200)
  Jul 31 21:52:36 nibz-falco-dev falco: 21:52:35.614809088: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=kube-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=15s resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:36.556668928: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:39.623470080: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:42.690744064: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:45.761656064: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:48.833193984: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:51.915211008: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:55.004199936: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:52:58.079348992: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)
  Jul 31 21:53:03 nibz-falco-dev falco: 21:53:01.160041984: Warning K8s Operation performed by user not in allowed list of users (user=br3dsptd0mfheg0375g0-admin target=cloud-controller-manager/endpoints verb=update uri=/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager resp=200)

Step 3. Set up forwarding of events from Falco into LogDNA

Then at last set up IBM Log Analysis with Log DNA on IBM Cloud. You want at least 30 days of retention for this example. The following screen capture shows how the Logging page within IBM Cloud might look after you provision an instance of the IBM Log Analysis with LogDNA service.

On the Logging tab, press your new log instance to Edit Log Sources. Selecting the Ubuntu/Debian Linux tab is the best way to locate the LogDNA API key and Log Host. Copy information about the key and the host. Notice that the host of the API and the log host vary. For this exercise, you just need the Log Host.

Set up an environment file or export these variables to your shell environment.

# Modify the following URL if your Log host is not in us-south
export LOGDNA_URL="https://logs.us-south.logging.cloud.ibm.com/logs/ingest"
export LOGDNA_KEY="eb250a1fedd547d6ae0a"

Now pull down the falco-logdna exporter script.

git clone https://github.com/falcosecurity/evolution
cp -r evolution/integrations/logdna/ .
virtualenv --python=python3 venv
source venv/bin/activate
pip install -r requirements.txt

In most cases, you must relax permissions on the Falco UNIX socket. You can also use a UNIX group if 777 is too insecure for you.

sudo chmod 777 /var/run/falco.sock

Now you can run the falco-logdna exporter.

python falco-logdna.py --logdna-key ${LOGDNA_KEY} --logdna-url ${LOGDNA_URL}

Return to the Logging page and click View LogDNA to open the LogDNA UI. You should see Kubernetes security events in the LogDNA UI such as the ones in the following screen capture.

Conclusion

1. You've done the following activities in this tutorial:

2. IBM Server Kubernetes Database setup to send audit logs to the service.

3. Set up a VM with correct networking and setup on an IBM Cloud VPC.

4. Set Falco up to accept payloads from Kubernetes for audit.

5. Configured Falco to store certain logs in LogDNA for further review

Tutorial: Host a website for a small company on IBM Cloud

Friday, 18 December 2020

Real-time auditing by Kubernetes with Falco