This will probably not be useful post for many but if you ever run into this post and it’s useful, let me know!
This time I had to get Guardian VM installed on OCI for our OT network to monitor the devices.
What is OT network?
An operational technology (OT) network refers to the hardware and software systems used to monitor and control physical processes, devices, and infrastructure in industrial settings. These systems are crucial for tasks like monitoring critical infrastructure, controlling robots in manufacturing, and managing energy grids. Unlike IT networks, OT networks prioritize real-time operation, safety, and reliability.
If you think from the OCI side, it’s a network which is kept isolated from the other OCI network VCNs and operations.
Installing Guardian sounded trivial but turned out to be slightly more challenging as expected, partly due to my lack of experience with Guardian and also first time setting up VTAPs in OCI.
Guardian VM Setup
First challenge was that the Guardian VM is not available in OCI, the good thing here is that it’s available as .OVA image so it’s importable to OCI. Once I had the image available on my own laptop, I followed standard process to import custom image to OCI you can find here.
Once the image was available, I created new instance in the correct subnet, no specific changes required apart from not setting the SSH key.
After instance was up and running, I did add secondary VNIC for the MONITORING traffic. This is where we will send the traffic. Note that even when OCI assigns IP to this VNIC, we should NOT assign any IP to it on the Guardian server Console.
Also, do NOT check the preserve source/destination IP, leave it unchecked.
The virtual Guardian installation instructions are here:
Once the VNIC was added, I logged to the Guardian server using OCI Console Connection for the instance and did the initial setup of adding admin password. Console Connection is useful feature to have available IMO.
NOTE: Since the second VNIC is added “on the fly”, I think it’s best to run factoryreset to the image so it discovers both VNICs during bootup, it might work without but I found this to be solution to get traffic showing up with Guardian in the end.
After you’re logged in with elevated privileges you can utilize command n2os-fullfactoryreset to do the reset.
After reboot you will need to reset admin credentials, and run setup command. During setup, check that your primary (MGMT) interface uses DHCP and/or has the IP you need. It can be vtnet0 or vtnet1.
For the MONITORING interface, make sure it’s enabled but DO NOT ASSIGN IP for it on the Guardian setup.
That’s it and that’s all for Guardian server side setup.
OCI VTAP and NLB Configuration
The interesting part comes here, how do we monitor the traffic from the servers as typically Guardian monitors traffic directly on the switches. Answer is OCI VTAP (Virtual Test Access Point), it will mirror all the traffic from a given VNIC to destination. Destination currently has to be OCI Network Load Balancer (NLB).
What is VTAP?
A Virtual Test Access Point (VTAP) provides a way to mirror traffic from a selected source to a selected target to help in troubleshooting, security analysis, and data monitoring.
Looking on the design and having multiple VCNs I established following:
- We need a Capture Filter assigned to VTAPs which logs ALL the traffic ingress and egress
- Each VNIC needs it’s own VTAP (unfortunately)
- We need to have a single NLB per VCN which forwards traffic to our Guardian MONITORING interface
- NLB Listener must be UDP Listener on port 4789
- NLB Listener must disable SRC/DST Header
- NLB Backend port 4789
- NLB Backend UNCHECK Preserve Source IP
- NLB Backend check ENABLE Fail Open (for traffic to be send to a failed health check destination)
Some generic comments from above.
The NLB Listener set to UDP port 4789 as per RFC 7348. I couldn’t find it on any documentation, there are couple links which mention it, not sure if something is missing or I’m not reading it right.
https://www.ateam-oracle.com/post/mirroring-traffic-in-oci-using-vtap
The other good information was found from Ali’s blog post above, you need to leave the preserve of source/destination unchecked/disabled for everything. You might think it’s needed but VTAP is handling that.
Thank you for both of the people writing above posts, really helpful!
Finally for the health check in the network load balancer, as the MONITORING interface doesn’t have any port that is active and listening, you can configure Fail Open mode in Backend so traffic is send to an unhealthy backend.
With above, our final design looked like this:
Summary
After doing above, we were able to see traffic in Guardian successfully. This shows that OCI does support variety of different requirements even for edge networking cases such as this.
Hopefully this helps someone one day!