Pipy and BPF: Creating a Non-intrusive and Transparent Traffic Interception Solution

Background

In the contemporary field of internet technology, microservice architecture has become the mainstream mode of application deployment. Although it enhances the agility of the development process and the scalability of the system, as the number of services increases, the system’s internal communication becomes increasingly complex. Especially when the interfaces and dependencies between services are opaque, the maintenance and optimization of the entire system become exceptionally difficult.

The traffic interception technology introduced today essentially provides a “god’s eye view” to observe and analyze the data flowing in complex network systems. This method allows development and operations teams to “peel the onion” of usually hidden and opaque service interactions, revealing detailed information and interaction patterns hidden in the data flow. With this high level of visualization and analytical capability, teams can deeply understand the operating mechanisms of services, even if these services appear to be closed and opaque “black boxes” on the surface.

Solution

The upgrade of Pipy to version 1.0 signifies its transformation from a mere data stream proxy to a comprehensive programmable application engine. This important evolution not only expands Pipy’s use cases but also significantly enhances its value and flexibility in modern network architectures. Using Pipy as a controller for BPF (Berkeley Packet Filter) is a typical example of its capability as a control plane. The addition of this capability provides a more powerful and flexible tool for network traffic management and system monitoring.

This solution utilizes the Pipy proxy as a control plane to load BPF (Berkeley Packet Filter) programs on host nodes, dynamically intercepting and forwarding target traffic based on preset configurations (such as port mapping relationships). In this process, traffic accessing the originalPort will be forwarded to a proxy service listening on a specified port. The proxy service further decodes and structures the packets according to the traffic’s protocol type, then sends the parsed request content through the HTTP protocol to storage like Elasticsearch for storage and analysis. Additionally, the traffic will be forwarded to the original target port unchanged, ensuring the continuity and transparency of the service.

Next, we demonstrate how to use Pipy for HTTP and Dubbo traffic interception and analysis.

Demonstration

In this demonstration, we will use the project traffic-interceptor, which contains three parts:

• rest2dubbo: Uses Pipy for HTTP to Dubbo protocol conversion.
• port-interceptor: A port interception controller implemented with Pipy, using BPF to redirect traffic accessing certain originalPorts to a specified port for traffic interception.
• traffic-dump: A proxy implemented with Pipy, listening on a specified port and then forwarding traffic to the originalPort, while also structuring the traffic information and sending it to Elasticsearch.

Below is the architecture diagram of the demonstration:

Prerequisites

• Ubuntu 22.04 (kernel version 6.5): The host running Dubbo and HTTP applications
• Client host (system type not limited): For running the rest2dubbo proxy and initiating HTTP requests
• Download and install Pipy 1.0.0
• Docker
• jq

Environment Setup

1. Start the Dubbo application, service on port 20880.

docker run --rm -d --name bookwarehouse -e spring.profiles.active=dubbo,dev -e dubbo.registry.register=false -p 20880:20880 addozhang/bookwarehouse-dubbo:0.3.1

1. Start the HTTP application.

pipy "pipy.listen(8000, $ => $.serveHTTP(new Message('Hi, Pipy')))"

1. Start Elasticsearch.

docker run --rm -d --name es -p 9200:9200 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.1

Environment Verification

On the client host, start the rest2dump proxy. This proxy will receive HTTP requests, convert them to Dubbo requests through protocol conversion, and forward them to the Dubbo application above.

git clone https://github.com/flomesh-io/pipy-demos.git
cd pipy-demos/traffic-interceptor

Before starting the proxy, you need to change the upstream service address in rest2dubbo/main.jsto the address of the host running the Dubbo service, for example, the host address I'm using is 20.24.215.69. Then start the proxy.

pipy rest2dubbo/main.js

After starting, you can see from the logs that it listens on port 8888. Use curl to send a request.

curl 'http://127.0.0.1:8888/v1/getBook' --header 'Content-Type: application/json' --data  '{"id": 1}'

You will receive the following response. This indicates our rest2dump protocol conversion proxy is running normally.

{
  "date": "2024-03-14T15:18:56.982Z",
  "isbn": "9787517054221",
  "author": "Flomesh",
  "name": "Introduction to Pipy",
  "value": 0,
  "id": "1"
}

Similarly, test the access to the HTTP application.

curl http://20.24.215.69:8000/hi

You should receive a normal response.

Hi, Pipy

Now we can test the traffic interception by starting the Traffic Dump proxy.

Starting Traffic Dump Proxy

Running the Traffic Dump proxy is simple, execute the following command.

git clone https://github.com/flomesh-io/pipy-demos.git
cd pipy-demos/traffic-interceptor
pipy traffic-dump/main.js

After running, you can see from the logs that it listens on ports 9000 and 30880.

Starting Port Interception

First, we need to compile the BPF program.

cd port-interceptor
make

In the bin directory, you can see the compiled BPF program port-interceptor.o.

Execute the following command to start port interception.

sudo pipy /main.js

Note: This script requires sudo because it needs to invoke tc to attach the BPF program to the kernel's data path, which requires administrator privileges.

After starting, you can see logs of Pipy updating BPF maps and loading the BPF program.

2024-03-14 15:48:11.881 [INF] Updating BPF maps...
2024-03-14 15:48:11.881 [INF]   Created port mapping 8000 <---> 9000
2024-03-14 15:48:11.881 [INF]   Created port mapping 8443 <---> 9443
2024-03-14 15:48:11.881 [INF]   Created port mapping 20880 <---> 30880

Verification

Resend the above HTTP and Dubbo requests, and both should receive normal responses.

curl http://20.24.215.69:8000/hi
curl 'http://127.0.0.1:8888/v1/getBook' --header 'Content-Type: application/json' --data  '{"id": 1}'

Access Elasticsearch to query saved request records.

HTTP Request:

curl -s -X GET "http://localhost:9200/http/_search" -H 'Content-Type: application/json' | jq .hits.hits
[
  {
    "_index": "http",
    "_type": "_doc",
    "_id": "W9mqPY4Bi_IvwOBjFiex",
    "_score": 1,
    "_source": {
      "time": 1710431540691,
      "host": "20.24.215.69:8000",
      "path": "/hi",
      "headers": "{\"host\":\"20.24.215.69:8000\",\"user-agent\":\"curl/8.4.0\",\"accept\":\"*/*\"}"
    }
  }
]

Dubbo Request:

curl -s -X GET "http://localhost:9200/dubbo/_search" -H 'Content-Type: application/json' | jq .hits.hits
[
  {
    "_index": "dubbo",
    "_type": "_doc",
    "_id": "WtmqPY4Bi_IvwOBjEycL",
    "_score": 1,
    "_source": {
      "time": 1710431539513,
      "dubboVer": "2.0.2",
      "interface": "io.flomesh.demo.api.BookWarehouseService",
      "ver": "v1",
      "method": "getBook",
      "args": [
        "Ljava/lang/String;",
        "1"
      ],
      "raw": "[\"2.0.2\",\"io.flomesh.demo.api.BookWarehouseService\",\"v1\",\"getBook\",\"Ljava/lang/String;\",\"1\",{\"kind\":\"map\",\"elements\":[[\"input\",\"196\"],[\"path\",\"io.flomesh.demo.api.BookWarehouseService\"],[\"interface\",\"io.flomesh.demo.api.BookstoreService\"],[\"version\",\"v1\"]]}]"
    }
  }
]

Bingo! The demonstration is complete.

Conclusion

Through this series of steps, we have shown how to use Pipy for traffic interception and analysis, all operations completed under the premise of not needing to alter the existing network structure and application code.

The release of Pipy 1.0 not only signifies its transformation from a data plane proxy to a programmable application engine but also heralds its significant role expansion in modern network architectures. Pipy’s high programmability and flexibility make it a powerful tool for implementing fine-grained traffic management and deep network analysis. Whether it’s enhancing network transparency, optimizing service performance, or strengthening security protection, Pipy can provide effective support.

<hr /><p>Pipy and BPF: Creating a Non-intrusive and Transparent Traffic Interception Solution was originally published in Flomesh on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>