Okay! I haven’t posted anything here for a while, but wanted to keep an effort of making semi-regular updates while I’m out and about. There’s lots more to say, but I’ll try to stick to just the major points.
In my previous post, I mentioned that I had bought my trailer in Vancouver. Only after purchasing it did I realize that I might hit some challenges around importing it into the States. I’m not the best at planning these kinds of things… but it all worked out in the end!
And now, for something completely unrelated to networking:
In August 2019, I bought a 16 foot fiberglass trailer – a 1977 Boler, to be specific – with the idea of fixing it up and living out of it for a big part of 2020. I’m hoping to use this blog to chronicle the entire build/remodel process, and then cover my adventures in it.
The Internet connection in Bokondini is a 1 Mbps satellite connection with a 10:1 contention ratio, which means that actual speeds can get as low as 100 Kbps. We haven’t been to Bok since we turned on the network, but when we were there, we had pretty much unlimited access for the three of us, with a request to use it lightly during school-hours. Even though the connection was a wee bit slower than our 100 Mbps line back home, I grew to really, really appreciate it, and am half inclined to think it’s the perfect amount of Internet. When I talk about our work in Bokondini I often get the question of “what can you do with 1 Mbps today anyways?” so I figured I’d write a post explicitly talking about some of our network experiences.
Older Protocols Work Better
First and foremost, older Internet protocols were inherently designed for slower speeds and less-reliable network conditions – so it’s no surprise they tend to work better. Somewhat interestingly, this split maps almost exactly to whether the protocol is designed to be run in the browser, or exists as a standalone application. Load up OSX’s “Mail” app that uses SMTP? No problem at all. Try to load Gmail in my browser? Not a chance!
On this note, a surprising culprit here was Slack. I naively assumed that the Slack app on my computer used XMPP, or any one of those classic standardized chat applications – but I was dead wrong. Turns out, Slack is Web-based, not that resilient, and surprisingly network-intensive for just a chat application. After realizing that I was literally unable to connect to Slack, I switched over all my correspondence to email, which went much, much smoother.
Web Interactivity Is Non-Existent
Our position behind a satellite link means that every round-trip takes a LOT longer than normal. This is a major killer for the new trend in modern (web 2.0? 3.0?) web-applications to (1) quickly load a shell of a page and (2) dynamically request the rest of the page as you interact with it. The biggest offenders of this practice that I noticed are articles that only load the next paragraph when you scroll down, rather than just fetching the whole page (all 20 kb of text, really?) and letting your browser handle the rendering.
Text-Only Webpages Are Awesome!
On the opposite end of the spectrum, we discovered certain news organizations provide stripped-down and text-only versions of their sites! If you’re curious, check out https://text.npr.org and https://lite.cnn.com for some great examples. It made me immensely happy to see content providers showing awareness of constrained-bandwidth scenarios, and is also worth mentioning that (to my understanding) these types of websites are much more friendly and compatible with accessibility tools (such as text-to-speech readers). We had to kinda “guess around” and use google to find some of these sites, but I’d love to see this practice become more widespread and/or standardized. It brings to mind how mobile website encoding started with the “m.” subdomain (still used by Wikipedia and pretty much no one else?) and then evolving into the responsive designs in use today… it would be fascinating to see backhaul-constrained access go the same way, maybe loading bare-bones content first and keying the responsivity off of network performance metrics.
CDNs Matter A Lot
Almost all satellite networks today are “bent-pipes,” meaning that the satellite doesn’t do any routing/switching and simply re-transmits our signal down to a predetermined ground station. In our case, that station was located in Australia, which means that websites with a CDN presence in Australia performed much, much better than US-based websites. Special shout-out to YouTube: I seriously didn’t expect video-streaming to work at all, but our YouTube tests came out remarkably usable, with relatively minimal buffering.
Interruptions are Interrupted
My favorite thing, by far, about working in Bokondini was that it felt like the Internet gave me everything I needed, but nothing I didn’t. The majority of my online work-tools (i.e. email and git) worked okay, if a little bit slower than usual, and my local work tools (text and code editors) were obviously fine. Even Google Docs worked great – I’d set it to offline mode, type my heart out, and then just get online to sync the document every so often.
Most general-use Web pages loaded fast enough to not be obnoxious, but slow enough to require my intent and sustained interest (relevant XKCD). Say I was writing an article and needed to lookup a statistic? Go to Google, search for it, wait a little bit, open the website, wait a little bit, not a problem at all. Unconsciously open a tab and surf over to Facebook to kill time? It loaded slowly enough to allow me some time to question/reflect on what I’m doing (really just wasting time), and I’d often end up closing the tab before it loaded. Turns out, the Internet’s a whole lot less of a distraction-machine when you make the instant gratification even slightly less instant!
1 Mbps For All?
Overall, I was remarkably impressed by how useful 1 Mbps Internet was, even in the moden Web ecosystem. Lately, I’ve been considering conducting another 1 Mbps experiment here in Seattle, to see if my user behavior comes out the same, or if the experience is changed by being geographically closer to common websites (or just being located in a highly connected city). Especially if it ends up helping me ignore my typical time-wasting sites, maybe it’ll turn out that more is less, less is more, and I end up preferring 1 Mbps to my 100!
Taking this idea even further, my colleague and friend Steve Song wrote a great post a while ago about the hypothetical economics of just providing free 2G cellular services (voice and SMS) to everyone. It’s pretty hard to top the social utility provided by voice and text services, but as more and more voice traffic turns into VoIP, I’ve started thinking and wondering about the Internet equivalent. Maybe ”1 Mbps for all” is really what we want to shoot for…
All right! Where I left off last time, we had just built the network, turned it off, and headed back to the States (just in time for another round of pitches and conferences). Once back in Seattle, we expected a relatively quick roll-out, but ended up having to pump the brakes while some regulatory questions got sorted out (and David returned to Papua from the States).
To my understanding, the terms of our license require that we not compete with any existing commercial telecom operator. In Bokondini, there exists a single operator, Telkomsel, who offers 2G coverage only. While the law clearly stated that we couldn’t compete with Telkomsel, “compete” was less clearly defined. Are we another cellular network? Yes. Do we offer 2G coverage? No. Do we offer telecommunications services (i.e. voice or text)? No. Could people use our network for telecommunications services? Via WhatsApp, absolutely. This set of questions ended up being our initial foray into the much larger existential question of “are we a telco or an ISP?” Or, more succinctly, “what the hell type of service do we provide?”
Initial Network Roll-Out:
On October 18, David sent us a WhatsApp message telling us that he had turned the network on in Bokondini and that everything looked good! Turns out, the system was already working as intended: he had traveled to Bokondini, powered everything on, inserted a SIM card, used the network to video-chat with his father in Florida for a hour, and then thought to give us a message. Two days later, we received word that he had distributed 10 SIM cards into the network and sold the first of many data packages to Fadly for resale. Without even realizing it, we were now a live business generating revenue!
Bugs, Bugs, and More Bugs:
I wish I could tell you the story ended there, but I’d be lying. As soon as we added our first ten users, the system started crashing, sometimes as often as every thirty seconds! This was, as you might expect, not the intended system behavior. After a week of frantic work, I was able to pin the issue down to a specific user’s phone, and stabilized the network by sending word to Bokondini kindly asking this user to please keep their phone off until further notice. Another two weeks of analyzing log files eventually revealed the culprit: a single incorrectly parsed header field, deep in our code, that was crashing our system every time the phone tried to join the network. A one-line fix and we were back in business… until we added another ten users and everything started crashing again! We stayed in this holding pattern for approximately two to three months: add ten users, brace for more issues, frantically fix them, take a day off, add ten more users. A grueling process, to be sure, but a necessary step on our march towards a stable LTE platform.
Interestingly, the majority of the bugs we flushed out had to do primarily with diversity in handsets. Phone manufacturers vary wildly in how (or if) they support certain fields and options, and different regions tend to see different manufacturers (For example, our customers predominantly use Oppo and Xiaomi phones, neither of which market their products in the United States). To make even more complex, LTE provides a large number of optional header fields and many different auth/ID workflows, which ends up creating a very wide range of corner-cases for our EPC to support.
End-of-the-Year Recap and 2019 Goals:
Though I didn’t know if we’d make it this far (or if I’d ever stop coding)… I’m proud to report that by the time December hit, we finally had stable, useful, functional LTE network running in Bokondini. I very seldom get bug reports now – and when I do, they’re predominantly fixed by something simple/stupid on my part. Most recently, I accidentally filled up the whole disk by enabling some basic logging tools to see how much traffic was being sent locally.
Moving forward into 2019, we’ve got a wide range of things we want to do, features we want to build, and research questions that we can finally start asking and answering. Now that the network stays running, our top engineering goals are code-cleaning, a couple feature-adds, and streamlining the deployment and configuration process, ideally to the point where a non-expert can download the project and get started without having to coordinate with us.
Research questions and curiosities range from the technical (how congested is the satellite link, how many users can we support?) to the economic (is this network profitable, how is it affecting the development of this community?) to the social (what are our users doing on the network?) to the personal (how much more fieldwork can I put into this project before my girlfriend dumps me?)
Obviously, these questions heavily intersect and interplay with each other. If we better understand what our users are doing on the network (social), we can build specific tools to support those actions (technical) which will undoubtably impact the overall usefulness of the network (economic). We’re hoping to really start digging into some of this research soon, especially with regards to using locally-hosted services as a way to relieve network congestion. As always, thanks for reading, and stay tuned for more!
First and foremost: a sincere apology to everyone following along! After we left for Papua, everything picked up a lot of steam and was a bit uncertain, and the pace didn’t let up until the end of the year. It’s really a shame I didn’t live-blog the process, but in the next posts I’ll try to re-cap the back half of the year. In 2019, I’m hoping to do better, especially since so many people have reached out to me about the blog asking to know how it went!
Getting To Bokondini
If I had to describe Papua in a single word, it would have to be “remote.” It took our team four separate flights over 3-4 days to arrive in Wamena, the major city with an airport in the Baliem Valley region. From there, we met up with Ibel and Helga, who helped us buy our necessary foodstuffs (food for three people for three weeks is no joke), navigate a somewhat convoluted visa process, and figure out the way to Bokondini – a three hour truck ride through muddy off-road terrain, including a river crossing. Making all of these connections while lugging around approximately 150 pounds of telecom equipment was a comical process, to say the least, but we (and the gear) made it safe and sound to Bokondini without any incidents to speak of.
In Bokondini, we stayed in a beautiful house owned by Scotty and Heidi Wisley. The Wisleys are a missionary couple who’ve been working in Bokondini for the past ten years, building out community leadership, infrastructure, and a local school. As it turns out, Scotty and Heidi were in Malaysia at a conference when we were in town, and they were gracious enough to lend us the use of their home during their absence.
Context Dictates Constraints
Even having spent a lot of time in various off-grid locations, I was particularly impressed by the infrastructural situation in Bokondini: a wild, anachronistic, perfect mix of old and new technologies, high-tech solutions paired with rural ingenuity and know-how.
Power to the village is provided by a combination of a solar array mounted on the schoolhouse roof and a small hydroelectric generator hooked up to a nearby stream, both inputs are wired to a bank of car-batteries. From the batteries, 12VDC is converted to 220VAC via a charge controller, and distributed throughout the village via wires hung from house to house in what can best be described as a nano-grid.
For purposes of power cleaning (as well as ensuring significant power for essential functions), the power is turned off daily from approximately 9pm to 6am. Given that the solar panels are already paid for and owned entirely by the community, the system works without any meters or billing – though during times with lower power generation, the news spreads through the community, and members are informally asked to conserve power (i.e. be sparing with their use of stoves and hot water).
There seemed to be no water infrastructure at all, micro or otherwise. Scotty and Heidi’s house is completely self-sufficient, and the taps draw unfiltered water from two main sources: a nearby river, or a rainwater catchment system. We then either boiled the water or ran it through a large, ten-liter Katadyn filter for drinking.
When we arrived, Bokondini already had very limited Internet access, via a 1Mbps satellite link. This link came into the main “technical room” above the schoolhouse, at which point it was turned into a WiFi hotspot, but restricted to the teachers in the community to do educationally things only. Informally, we were told that one of the main drivers of network traffic was teachers using YouTube as an educational resource – more on this below.
Mounting The Tower
Once we were settled in, the work began in earnest. In a nutshell, the main effort was to lower a pole-tower, remove the (unused) dish on the top, replace it with our LTE gear and antennas, and secure the pole back the way it was. Once mounted, we would connect our EPC to the gear on the tower, power the whole thing on, and (hopefully) have a small-scale LTE network that would cover the entire community.
Mounting the gear on the tower was no small feat. Thankfully, with help and guidance from Ibel, who knows the infrastructure in Bokondini like the back of his hand, we got the job done, and done well. Twice a fast-moving thunderstorm chased us off the corrugated roof, and even after the work was done, we had to pull the tower down to replace a malfunctioning eNodeB… but ten days after we arrived, we finally raised the tower with our gear for the last time.
Meanwhile, when we weren’t climbing up on rooftops, we were hard at work configuring the EPC to stay online across a wide range of unorthodox (yet not uncommon) situations. Despite having verified our EPC’s functionality in a controlled lab setting, keeping it alive out in Bokondini required a substantial amount of in-the-field programming. The considerations ranged from the obvious (make sure the box turns itself back on after a power failure) to the subtle (ensuring that if one component crashes, all the correct other components reboot so that the network recovers into a functional and useful state). All in all, the process made for a fascinating crash-course in Keeping Systems Alive, though I’d be lying if I said I enjoyed the course while it was happening.
It’s well known that the best way to flush out bugs is to dogfood a product – so during this time, we aggressively dogfooded the network as hard as we could. Connect a phone, walk around town, run a hotspot back at the house all day – all the while, obsessively looking for any network disconnects or stumbles. One of the most noteworthy bugs seemed to be triggered by a specific phone in the community that tried to connect to our network (and subsequently crashed it) every day, once a day, around 3pm. While the fix was anticlimactic (a single incorrectly-parsed header field), the workflow was hilarious: dig through the logs, make a guess, try to write a fix… and then wait for 3pm the next day to see if it worked.
Finally, on our second-to-last day, we had a stable network: No crashes, no bugs, no stumbles: just reliable Internet, from lights-on to lights-off. At this point, we hadn’t yet distributed any SIM cards for a number of reasons, both regulatory and social. We turned off the network to save power, left some instructions for turning it back on (just plug it in), and hit the road, for another three day’s worth of flights back to Seattle – cautiously optimistic about the impending network launch, but without much of a clue what to expect.
Okay, that’s a wrap for now. In Part 2 I’ll cover the initial network launch, the process of bringing additional phones/users online, and some of our initial results of how the network’s doing. Spoiler alert: It’s going really, really well! 🙂
My colleague Matt Johnson and I got featured on the Internet Society Blog for our work in Bokondini! Reprinted in full below, but you can access the original copy here: https://www.internetsociety.org/blog/2018/09/building-a-community-lte-network-in-bokondini-indonesia/
Building a Community LTE Network in Bokondini, Indonesia
As part of our work in the University of Washington’s Information and Communications Technology for Development (ICTD) Lab, we recently spent four weeks in Bokondini, a village in the Papuan Highlands. During our time in Bokondini, we helped some community members extend Internet access throughout the village via a community LTE network, using a technology stack that we call CoLTE (for Community LTE).
The Area and Background
Bokondini is a small village (population ~1,500) in the Baliem Valley, a mountainous region located in the highlands of Indonesian Papua. The Papuan Highlands are a famously rugged, remote, and hard-to-cover area, and many inhabitants of the region live without any form of telecommunications whatsoever. Infrastructure in Bokondini is a remarkably ad-hoc process; for example, electricity comes from a small set of solar panels and a micro hydro generator, and tends to shut off between the hours of 9pm and 6am.
Bokondini’s current relationship to the Internet revolves primarily around the local school. The community pays for a small (1Mbps) satellite Internet connection that terminates at the local elementary school, where it’s used to provide WiFi coverage to teachers on the school campus. Coverage is extended to a few other houses in the community via directional antennas and WiFi hotspots, but the vast majority of community members do not have Internet access.
The Internet connection is managed and supported by Airwave Missions, a Wireless ISP (WISP) based in Wamena (Wamena is the main town in the Baliem Valley, and is approximately 3 hours away from Bokondini by truck) and run by David Haag. Airwave Missions is a textbook example of rural know-how and problem-solving, combining skilled technical knowledge and local community relationships to provide Internet coverage to many of the small, hard-to-reach villages throughout the valley. Every site that Airwave Missions serves is unique, and tends to require a custom, and often creative, solution, typically involving many different technologies and solutions. These technologies include the standard WISP set of satellite links, directional wireless links, and different forms of power (both on- and off-grid), but have gotten as creative as a custom-built lightweight, solar-powered wireless repeater that they installed on a mountaintop using a helicopter!
Airwave Missions’ fieldwork operations are supported by a classic jack-of-all-trades technical support person named Ibel. Ibel has a deep understanding of all forms of rural infrastructure, and is equally comfortable fixing a DC hydroelectric generator with a multimeter as he is climbing a tree to anchor a guyline, lowering and hoisting an antenna tower, or wrenching on a two-stroke motorcycle. Ibel was our go-to support throughout the entire CoLTE installation – not just figuring out a wide range of issues, but really supporting and grounding the project within the community itself.
Our team’s relationship to Bokondini started approximately five years ago, when Professor Kurtis Heimerl (then a graduate student at UC Berkeley) was contacted by a community member, and traveled there to help build a 2G community cellular network. The network was a great success, and was operational for over five years, until it was shut down in Spring 2018 as a result of a national telecom company expanding 2G voice and SMS coverage to Bokondini. The success of our previous 2G pilot, in combination with the lack of broad Internet access, led us to believe that a small-scale LTE network could be a viable solution to cover the entire community all at once, rather than continue to connect individual houses via directional antennas and WiFi hotspots.
Our team spent September 2018 to June 2019 juggling a wide range of tasks: figuring out LTE, exploring our different open-source options, building CoLTE, ordering equipment (base stations, test handsets, and antennas), printing SIM cards, and coordinating with Airwave Missions on a wide range of logistics, from frequencies and geographic coverage to more mundane challenges such as lodging. In July, we packed all the equipment into three large containers and flew it all out to Wamena, where we met up with Ibel and trucked everything out to Bokondini.
Once in Bokondini, the entire installation process spanned a couple of weeks, during which the work took on a wide range of forms. We encountered unexpected equipment failures, replaced one of our US-made switches that only ran on 110VAC (Indonesian grid power is 220VAC), spent some time furiously debugging OpenAirInterface, and had to get a bit creative with the antenna mounting process – but the speed bumps were minor and manageable, and all-in-all the deployment went about as smoothly as could be expected.
Results and Next Steps
Once the network was up and working reliably, we all spent a couple of days walking around Bokondini, collecting metrics on signal-strength, range, and network performance – quite literally, the old “can you hear me now?” commercial. The results were, in a single word, great. The network covers over a mile, and easily serves the entire community. WhatsApp messages work just fine, and WhatsApp voice calls come through at high quality, even if there’s a bit of delay (fully understandable – the signal gets bounced to space and back two separate times!). All websites seemed to work without any problem, even if they loaded slowly. The one outlier we found was that YouTube traffic loaded surprisingly quickly.
With the network built and tested, all that remains is to begin the public network launch. Airwave Missions is currently finalizing costs, distributing SIM cards, and creating a gradual deployment plan. We hope to bring the network live for the first batch of test users at the end of September, and then gradually roll out additional coverage.
While Airwaves Missions is rolling out coverage, the UW team returned to Seattle, and has been hard at work on several CoLTE-related fronts. These tasks include simplifying the installation front, hardening the code and system, adding web-based configuration and status-monitoring tools, and producing a basic how-to guide for anyone seeking to deploy their own community LTE network. For more information on CoLTE, or to get involved with a deployment, please email email@example.com.