After however many years working on open source LTE, I’ve recently been forced graciously invited to upgrade some of the networks I support from LTE to 5G. My thoughts on 5G as an architecture are enumerated in other posts, so I wanted to take this moment to discuss and provide something useful to my fellow engineers: A clean, somewhat-readable 5G PCAP to go along with the LTE ones I made so many years ago. Interestingly enough, just the process of obtaining a clean PCAP was challenging/complex enough to merit its own post.
Step 0: What The Hell Is Going On?
As the type of person who learns best by getting my hands on stuff and making it work, my golden standard for how I teach myself something new is logs (often unreliable) and PCAPs (almost always reliable). As an additional benefit, countless buzzwords and acronyms melt as soon as you start asking “so what exactly goes over the wire? Can I see a PCAP of it?” and by now I basically consider a new technology vaporware until I see a packet capture indicating it actually exists. Consider it the equivalent of requiring “photographic evidence“ for any new species of cryptid.1
To collect my first 5G pcap, I installed and ran all the open5gs 5G functions on a single computer, connected an Open-RAN gNB to the core, and then connected a handset. I expected a capture that looked at least somewhat familiar to my prior experience in LTE, maybe somewhere on the order of 15-25 messages… and instead, opened up a log containing almost a thousand packets. And to think I previously thought LTE was overly complicated and verbose!
Step 1: Getting My Bearings
The very first thing I noticed in looking at the PCAP was that all the communication within the 5G core is handled over HTTP2. This is… potentially a good choice, depending on exactly how the tasks and microservices are divided (we’ll have to wait and see), but it also bloats the number of packets to a pretty large degree. Any message passed between two services requires a TCP three way handshake as well as additional HTTP2-specific connection setup and management messages. Adding the Wireshark filter “http2 or ngap”2 immediately removed all the TCP functionality and dropped the number of messages in my capture to slightly over 400. 50% cut rate right out the gate, not so bad!
Step 2: Disable the Secure Communications Proxy
Digging further into the messages that remained, I noticed that basically every HTTP2 message containing actual content was duplicated. The first HTTP2 POST would always be from 127.0.0.1 to 127.0.0.200, and then the exact message would immediately repeat, this time from 127.0.0.1 -> 127.0.0.X, where 127.0.0.X appeared to be the actual destination. I dug through the config files and found the culprit was the Secure Communications Proxy (SCP), a 5G “service” that simply man-in-the-middles communications between 5G services with the explicit goal of masking the source IP address. This service could be somewhat useful in specific contexts (e.g. a network administered across multiple organizations) but for our purposes (an educational PCAP that helps us understand how 5G works) it just adds unnecessary noise and clutter. I disabled this function and effectively cut the number of messages in the PCAP down to 250. Still large, but a far cry from where we started!
Step 3: Add A Feature To Support Local CURL Binding
Once I disabled the SCP feature, I anticipated reading HTTP2 messages from one microservice to another and being able to loosely reason about what was going on. This was… half true. With the default open5gs configuration, each 5G microservice binds a different local IP address to make communications clear. For example, the AMF binds 127.0.0.5 and the SMF binds 127.0.0.4. However, any time one of these services sends a message to another service, it sets up a new socket and doesn’t bother binding a local address or port. This is a pretty standard practice in network programming, no notes from me there, but it unfortunately means that each outgoing message in a localhost context originates from 127.0.0.1. In short, I had a PCAP full of messages where I could easily understand the destination of the message, but not the source.
This bothered me enough that I dug through the libcurl documentation to figure out how to bind a specific local IP address when sending a HTTP request and then submitted a pull request adding this feature to open5gs. With this relatively obscure setting applied, outgoing HTTP requests from a specific microservice will always come from the specific IP address in the parameter. Binding the same IP address for both incoming and outgoing communication finally got me a PCAP that let me know who was talking to whom. Remarkably nontrivial, but hey – welcome to 5G!
Step 4: Filter Out NRF Resolution
I expected the aforementioned PCAP to read as easily as a well-written novel,3 but this was still not the case. Rather, I encountered a bunch of random and inconsistent communication that I eventually figured out were individual microservices resolving the IP address of another microservice. More specifically, the first time one service (e.g. the AMF) wants to talk to another service (e.g. the AUSF), it first sends a message to the NRF asking to resolve the address of the AUSF. The address is then cached and the AMF does not query the NRF for subsequent messages. This approach (and even some of the message fields involved) is very, very, very similar to DNS, just… completely re-written and re-standardized in a way they never bothered to explain well. Additionally, given that these name resolution messages are sent over HTTP2, there’s no super simple way to filter them out. Such is the life of a telco engineer.
My ham-fisted solution to this problem was to simply filter out all messages to or from the NRF. Assuming the stock open5gs configuration, this can be accomplished by adding “and ip.addr != 127.0.0.10” to the http2 filter. I would like to take this moment to note that if this was IP networking, you could have done this with a Wireshark filter like !dns or udp.port!=53, or you could have intuitively distinguished name resolution from actual data comms by the ports and protocols used, but novices like myself4 should refrain from questioning the wisdom of the 3GPP. For those of you keeping score at home, this move dropped the total number of packets down to 170.
Step 5: Filter Out HTTP2 Noise
One final step for kicks: the PCAP created by the above filter still had a fair amount of relatively irrelevant messages to our purpose. The “http2” filter already gets rid of TCP handshakes and ACKs, but HTTP2 itself has a wide range of specific connection setup and management messages, such as the “Magic” message sent at the beginning of every connection for reasons still not entirely clear to me. Empirically, adding “and !http2.magic and !http2.type==4 and !http2.type==8” to our HTTP2 filter got me to our final product, linked below and added to the PCAP Library: A clear, concise, and relatively easy to read reference PCAP of a network attach in 5G. I’ve trimmed out any messages that don’t directly correspond to 5G content being sent/received, and as a result every packet should have something relatively useful for the reader. Note that as a result of cropping unneeded TCP messages, Wireshark will flag some of the packets as acking a previous uncaptured segment but this has no impact on anything.
The PCAP clocks in at slightly over 100 messages, but almost half of those messages are HTTP2 successful responses – something along the lines of “200 OK” or “201 CREATED”. Not incredibly relevant or hard to read, but useful semantically and important to not filter out, because ideally any errors should show up as big red 400 or 404 responses. That being said, adding “and http2.header.name != ":status”” blanks out these messages and gets us to a final weigh-in of 43 packets. More verbose than LTE, but when we get down to the actual content being sent and received… not too much worse, either.
Now that I have something that’s small enough to read and reason about, I plan to dig through these messages and compare/contrast with LTE – but that’ll have to be a future post. More to come later and I hope someone finds the attached capture useful!
- I know Bigfoot is out there please don’t send me Anti-Cascadian propaganda disputing this obvious fact. ↩︎
- ngap is the over-the-wire protocol used between the core and the gNB and looks/feels quite similar to s1ap. ↩︎
- I recently finished No Country For Old Men if anyone’s looking for recommendations. ↩︎
- I literally have a PhD in wireless networking. ↩︎
