Netflix had earlier released a performance monitoring tool called Vector as open source. Vector can “visualize and analyze system and application-level metrics in near real-time”. These metrics include CPU, memory, disk and network, and application profiling using flamegraphs. Vector is build on top of Performance Co-Pilot (PCP), a performance analysis toolkit. PCP works in a distributed fashion with a daemon on each monitored host, which controls and routes metric requests to individual agents which collect the actual metrics. There are agents for most popular software, and custom application metrics can be collected by writing one’s own agent. Client applications connect to the daemon.
Vector adds an AngularJS and D3.js based UI on top of this which pulls performance data from a PCP web daemon which in turn talks to the metric collector daemon. This model is lightweight as no data is stored across browser sessions, and there is no metrics aggregation across hosts.
eBPF is a set of a low level APIs that make it easy to build profiling tools for the Linux kernel. To integrate eBPF tools with Vector, PCP support was developed for eBPF so that its output could be routed to Vector via the standard PCP architecture. Metrics collected include filesystem latency, I/O top processes, TCP session information, and kernel scheduler run queue latency – all of which can be viewed in Vector. eBPF has been leveraged by other tools to collect and monitor container metrics instead of using kernel modules.
Netflix has gradually moved many of its services to containers, which are managed by Titus – its container management platform. The increased use of containers led to more use cases to monitor them. Although container metrics support has been available on Vector for quite some time, its new release has changes so that container and host metrics can be analyzed together. There are also UI improvements which make it easier to resize charts, pause/resume graphs and to continue to collect data when the browser tab goes out of focus. Internally, the UI has switched to React and the Webpack build tool.