Bokondini Initial Roll-Out

All right! Where I left off last time, we had just built the network, turned it off, and headed back to the States (just in time for another round of pitches and conferences). Once back in Seattle, we expected a relatively quick roll-out, but ended up having to pump the brakes while some regulatory questions got sorted out (and David returned to Papua from the States).

To my understanding, the terms of our license require that we not compete with any existing commercial telecom operator. In Bokondini, there exists a single operator, Telkomsel, who offers 2G coverage only. While the law clearly stated that we couldn’t compete with Telkomsel, “compete” was less clearly defined. Are we another cellular network? Yes. Do we offer 2G coverage? No. Do we offer telecommunications services (i.e. voice or text)? No. Could people use our network for telecommunications services? Via WhatsApp, absolutely. This set of questions ended up being our initial foray into the much larger existential question of “are we a telco or an ISP?” Or, more succinctly, “what the hell type of service do we provide?”

Initial Network Roll-Out:

On October 18, David sent us a WhatsApp message telling us that he had turned the network on in Bokondini and that everything looked good! Turns out, the system was already working as intended: he had traveled to Bokondini, powered everything on, inserted a SIM card, used the network to video-chat with his father in Florida for a hour, and then thought to give us a message. Two days later, we received word that he had distributed 10 SIM cards into the network and sold the first of many data packages to Fadly for resale. Without even realizing it, we were now a live business generating revenue!

Bugs, Bugs, and More Bugs:

I wish I could tell you the story ended there, but I’d be lying. As soon as we added our first ten users, the system started crashing, sometimes as often as every thirty seconds! This was, as you might expect, not the intended system behavior. After a week of frantic work, I was able to pin the issue down to a specific user’s phone, and stabilized the network by sending word to Bokondini kindly asking this user to please keep their phone off until further notice. Another two weeks of analyzing log files eventually revealed the culprit: a single incorrectly parsed header field, deep in our code, that was crashing our system every time the phone tried to join the network. A one-line fix and we were back in business… until we added another ten users and everything started crashing again! We stayed in this holding pattern for approximately two to three months: add ten users, brace for more issues, frantically fix them, take a day off, add ten more users. A grueling process, to be sure, but a necessary step on our march towards a stable LTE platform.

Interestingly, the majority of the bugs we flushed out had to do primarily with diversity in handsets. Phone manufacturers vary wildly in how (or if) they support certain fields and options, and different regions tend to see different manufacturers (For example, our customers predominantly use Oppo and Xiaomi phones, neither of which market their products in the United States). To make even more complex, LTE provides a large number of optional header fields and many different auth/ID workflows, which ends up creating a very wide range of corner-cases for our EPC to support.

End-of-the-Year Recap and 2019 Goals:

Though I didn’t know if we’d make it this far (or if I’d ever stop coding)… I’m proud to report that by the time December hit, we finally had stable, useful, functional LTE network running in Bokondini. I very seldom get bug reports now – and when I do, they’re predominantly fixed by something simple/stupid on my part. Most recently, I accidentally filled up the whole disk by enabling some basic logging tools to see how much traffic was being sent locally.

Moving forward into 2019, we’ve got a wide range of things we want to do, features we want to build, and research questions that we can finally start asking and answering. Now that the network stays running, our top engineering goals are code-cleaning, a couple feature-adds, and streamlining the deployment and configuration process, ideally to the point where a non-expert can download the project and get started without having to coordinate with us.

Research questions and curiosities range from the technical (how congested is the satellite link, how many users can we support?) to the economic (is this network profitable, how is it affecting the development of this community?) to the social (what are our users doing on the network?) to the personal (how much more fieldwork can I put into this project before my girlfriend dumps me?)

Obviously, these questions heavily intersect and interplay with each other. If we better understand what our users are doing on the network (social), we can build specific tools to support those actions (technical) which will undoubtably impact the overall usefulness of the network (economic). We’re hoping to really start digging into some of this research soon, especially with regards to using locally-hosted services as a way to relieve network congestion. As always, thanks for reading, and stay tuned for more!

Our Time In Bokondini

First and foremost: a sincere apology to everyone following along! After we left for Papua, everything picked up a lot of steam and was a bit uncertain, and the pace didn’t let up until the end of the year. It’s really a shame I didn’t live-blog the process, but in the next posts I’ll try to re-cap the back half of the year. In 2019, I’m hoping to do better, especially since so many people have reached out to me about the blog asking to know how it went!

Getting To Bokondini

If I had to describe Papua in a single word, it would have to be “remote.” It took our team four separate flights over 3-4 days to arrive in Wamena, the major city with an airport in the Baliem Valley region. From there, we met up with Ibel and Helga, who helped us buy our necessary foodstuffs (food for three people for three weeks is no joke), navigate a somewhat convoluted visa process, and figure out the way to Bokondini – a three hour truck ride through muddy off-road terrain, including a river crossing. Making all of these connections while lugging around approximately 150 pounds of telecom equipment was a comical process, to say the least, but we (and the gear) made it safe and sound to Bokondini without any incidents to speak of.

In Bokondini, we stayed in a beautiful house owned by Scotty and Heidi Wisley. The Wisleys are a missionary couple who’ve been working in Bokondini for the past ten years, building out community leadership, infrastructure, and a local school. As it turns out, Scotty and Heidi were in Malaysia at a conference when we were in town, and they were gracious enough to lend us the use of their home during their absence.

Context Dictates Constraints

Even having spent a lot of time in various off-grid locations, I was particularly impressed by the infrastructural situation in Bokondini: a wild, anachronistic, perfect mix of old and new technologies, high-tech solutions paired with rural ingenuity and know-how.

Power:
Power to the village is provided by a combination of a solar array mounted on the schoolhouse roof and a small hydroelectric generator hooked up to a nearby stream, both inputs are wired to a bank of car-batteries. From the batteries, 12VDC is converted to 220VAC via a charge controller, and distributed throughout the village via wires hung from house to house in what can best be described as a nano-grid.

IMG_1315.JPG
Bokondini’s electrical system

For purposes of power cleaning (as well as ensuring significant power for essential functions), the power is turned off daily from approximately 9pm to 6am. Given that the solar panels are already paid for and owned entirely by the community, the system works without any meters or billing – though during times with lower power generation, the news spreads through the community, and members are informally asked to conserve power (i.e. be sparing with their use of stoves and hot water).

 

Water:
There seemed to be no water infrastructure at all, micro or otherwise. Scotty and Heidi’s house is completely self-sufficient, and the taps draw unfiltered water from two main sources: a nearby river, or a rainwater catchment system. We then either boiled the water or ran it through a large, ten-liter Katadyn filter for drinking.

IMG_1387.JPG
“Just hand them up, it’ll be fine”

Internet:
When we arrived, Bokondini already had very limited Internet access, via a 1Mbps satellite link. This link came into the main “technical room” above the schoolhouse, at which point it was turned into a WiFi hotspot, but restricted to the teachers in the community to do educationally things only. Informally, we were told that one of the main drivers of network traffic was teachers using YouTube as an educational resource – more on this below.

IMG_1418.JPG
Raising the tower

Mounting The Tower

Once we were settled in, the work began in earnest. In a nutshell, the main effort was to lower a pole-tower, remove the (unused) dish on the top, replace it with our LTE gear and antennas, and secure the pole back the way it was. Once mounted, we would connect our EPC to the gear on the tower, power the whole thing on, and (hopefully) have a small-scale LTE network that would cover the entire community.

Mounting the gear on the tower was no small feat. Thankfully, with help and guidance from Ibel, who knows the infrastructure in Bokondini like the back of his hand, we got the job done, and done well. Twice a fast-moving thunderstorm chased us off the corrugated roof, and even after the work was done, we had to pull the tower down to replace a malfunctioning eNodeB… but ten days after we arrived, we finally raised the tower with our gear for the last time.

Meanwhile, when we weren’t climbing up on rooftops, we were hard at work configuring the EPC to stay online across a wide range of unorthodox (yet not uncommon) situations. Despite having verified our EPC’s functionality in a controlled lab setting, keeping it alive out in Bokondini required a substantial amount of in-the-field programming. The considerations ranged from the obvious (make sure the box turns itself back on after a power failure) to the subtle (ensuring that if one component crashes, all the correct other components reboot so that the network recovers into a functional and useful state). All in all, the process made for a fascinating crash-course in Keeping Systems Alive, though I’d be lying if I said I enjoyed the course while it was happening.

IMG_1453.JPG
A job well done!

It’s well known that the best way to flush out bugs is to dogfood a product – so during this time, we aggressively dogfooded the network as hard as we could. Connect a phone, walk around town, run a hotspot back at the house all day – all the while, obsessively looking for any network disconnects or stumbles. One of the most noteworthy bugs seemed to be triggered by a specific phone in the community that tried to connect to our network (and subsequently crashed it) every day, once a day, around 3pm. While the fix was anticlimactic (a single incorrectly-parsed header field), the workflow was hilarious: dig through the logs, make a guess, try to write a fix… and then wait for 3pm the next day to see if it worked.

Finally, on our second-to-last day, we had a stable network: No crashes, no bugs, no stumbles: just reliable Internet, from lights-on to lights-off. At this point, we hadn’t yet distributed any SIM cards for a number of reasons, both regulatory and social. We turned off the network to save power, left some instructions for turning it back on (just plug it in), and hit the road, for another three day’s worth of flights back to Seattle – cautiously optimistic about the impending network launch, but without much of a clue what to expect.

Okay, that’s a wrap for now. In Part 2 I’ll cover the initial network launch, the process of bringing additional phones/users online, and some of our initial results of how the network’s doing. Spoiler alert: It’s going really, really well! 🙂

Guest Post on the Internet Society Blog

My colleague Matt Johnson and I got featured on the Internet Society Blog for our work in Bokondini! Reprinted in full below, but you can access the original copy here: https://www.internetsociety.org/blog/2018/09/building-a-community-lte-network-in-bokondini-indonesia/

Building a Community LTE Network in Bokondini, Indonesia

As part of our work in the University of Washington’s Information and Communications Technology for Development (ICTD) Lab, we recently spent four weeks in Bokondini, a village in the Papuan Highlands. During our time in Bokondini, we helped some community members extend Internet access throughout the village via a community LTE network, using a technology stack that we call CoLTE (for Community LTE).

The Area and Background

Bokondini is a small village (population ~1,500) in the Baliem Valley, a mountainous region located in the highlands of Indonesian Papua. The Papuan Highlands are a famously rugged, remote, and hard-to-cover area, and many inhabitants of the region live without any form of telecommunications whatsoever. Infrastructure in Bokondini is a remarkably ad-hoc process; for example, electricity comes from a small set of solar panels and a micro hydro generator, and tends to shut off between the hours of 9pm and 6am.

Bokondini’s current relationship to the Internet revolves primarily around the local school. The community pays for a small (1Mbps) satellite Internet connection that terminates at the local elementary school, where it’s used to provide WiFi coverage to teachers on the school campus. Coverage is extended to a few other houses in the community via directional antennas and WiFi hotspots, but the vast majority of community members do not have Internet access.

The Internet connection is managed and supported by Airwave Missions, a Wireless ISP (WISP) based in Wamena (Wamena is the main town in the Baliem Valley, and is approximately 3 hours away from Bokondini by truck) and run by David Haag. Airwave Missions is a textbook example of rural know-how and problem-solving, combining skilled technical knowledge and local community relationships to provide Internet coverage to many of the small, hard-to-reach villages throughout the valley. Every site that Airwave Missions serves is unique, and tends to require a custom, and often creative, solution, typically involving many different technologies and solutions. These technologies include the standard WISP set of satellite links, directional wireless links, and different forms of power (both on- and off-grid), but have gotten as creative as a custom-built lightweight, solar-powered wireless repeater that they installed on a mountaintop using a helicopter!

Airwave Missions’ fieldwork operations are supported by a classic jack-of-all-trades technical support person named Ibel. Ibel has a deep understanding of all forms of rural infrastructure, and is equally comfortable fixing a DC hydroelectric generator with a multimeter as he is climbing a tree to anchor a guyline, lowering and hoisting an antenna tower, or wrenching on a two-stroke motorcycle. Ibel was our go-to support throughout the entire CoLTE installation – not just figuring out a wide range of issues, but really supporting and grounding the project within the community itself.

Our team’s relationship to Bokondini started approximately five years ago, when Professor Kurtis Heimerl (then a graduate student at UC Berkeley) was contacted by a community member, and traveled there to help build a 2G community cellular network. The network was a great success, and was operational for over five years, until it was shut down in Spring 2018 as a result of a national telecom company expanding 2G voice and SMS coverage to Bokondini. The success of our previous 2G pilot, in combination with the lack of broad Internet access, led us to believe that a small-scale LTE network could be a viable solution to cover the entire community all at once, rather than continue to connect individual houses via directional antennas and WiFi hotspots.

The Project

Our team spent September 2018 to June 2019 juggling a wide range of tasks: figuring out LTE, exploring our different open-source options, building CoLTE, ordering equipment (base stations, test handsets, and antennas), printing SIM cards, and coordinating with Airwave Missions on a wide range of logistics, from frequencies and geographic coverage to more mundane challenges such as lodging. In July, we packed all the equipment into three large containers and flew it all out to Wamena, where we met up with Ibel and trucked everything out to Bokondini.

Once in Bokondini, the entire installation process spanned a couple of weeks, during which the work took on a wide range of forms. We encountered unexpected equipment failures, replaced one of our US-made switches that only ran on 110VAC (Indonesian grid power is 220VAC), spent some time furiously debugging OpenAirInterface, and had to get a bit creative with the antenna mounting process – but the speed bumps were minor and manageable, and all-in-all the deployment went about as smoothly as could be expected.

Results and Next Steps

Once the network was up and working reliably, we all spent a couple of days walking around Bokondini, collecting metrics on signal-strength, range, and network performance – quite literally, the old “can you hear me now?” commercial. The results were, in a single word, great. The network covers over a mile, and easily serves the entire community. WhatsApp messages work just fine, and WhatsApp voice calls come through at high quality, even if there’s a bit of delay (fully understandable – the signal gets bounced to space and back two separate times!). All websites seemed to work without any problem, even if they loaded slowly. The one outlier we found was that YouTube traffic loaded surprisingly quickly.

With the network built and tested, all that remains is to begin the public network launch. Airwave Missions is currently finalizing costs, distributing SIM cards, and creating a gradual deployment plan. We hope to bring the network live for the first batch of test users at the end of September, and then gradually roll out additional coverage.

While Airwaves Missions is rolling out coverage, the UW team returned to Seattle, and has been hard at work on several CoLTE-related fronts. These tasks include simplifying the installation front, hardening the code and system, adding web-based configuration and status-monitoring tools, and producing a basic how-to guide for anyone seeking to deploy their own community LTE network. For more information on CoLTE, or to get involved with a deployment, please email colte@cs.washington.edu.

CoLTE Won Third Place In The Mozilla WINS Challenge!

I’m thrilled to announce that CoLTE received third place in the Mozilla Wireless Innovations for a Networked Society challenge! The finalist competition was a super fun science-fair-esque bake off, full of different projects and working on bridging the digital divide and helping expand connectivity.

https://spectrum.ieee.org/view-from-the-valley/telecom/wireless/mozilla-and-nsf-hand-out-16-million-to-wireless-challenge-winners

 

Off To Indonesia

Hi everyone,

Today, the rubber hits the road! My partner, Audrey, and I spent all Saturday running last minute errands, sending last-minute email questions (“should I bring a camping stove or not?”) and packing all sorts of equipment into two rubbermaid totes and our personal bags. Packing’s been a comically awkward fusion of my outdoor life with my professional life, packing everything from tarps and mosquito nets and hiking boots to literally 110 pounds of sensitive electronic equipment, all crammed into two rubbermaid totes and our personal bags. We’ve got eNodeBs, EPCs, one thousand SIM cards, mounting hardware, antenna cables, ethernet cables and routers, extras of everything, a power strip, and all sorts of other tools, all *just barely* under the airline weight requirements and padded as best we could with styrofoam. At 11pm we lugged it all down to the airport, and at 2am we were off to Jakarta. If you check it out on a map, you’ll notice that Jakarta’s actually way out of the way from Bokondini (as in, 7 extra hours of flying), but Jakarta’s where international flights have to land in order to clear customs and enter Indonesia.

IMG_20180726_102342217.jpgWe met our teammate Matt (who flew in from fieldwork in the Philippines) at the airport, and after a day or two, the three of us will head into the field, with a small regional flight taking us from Jakarta to Jayapura, an even smaller flight from Jayapura to Wamena, and then finally a truck ride out to Bokondini. Once in Bokondini, our plan is to stay with a couple of Christian missionaries who have been working in the local community for the past twenty years or so. Scott and Heidi have been instrumental in helping us figure out what the on-the-ground logistics will be, what we’ll need to bring and do, not to mention opening their homes to us! We’re already incredibly grateful for their support, and we haven’t even arrived yet 🙂

Okay – well that wraps it up for this post, stay tuned for more content and pictures as we move through Indonesia and get further out into the jungle!

Money Matters (Part 3)

This post is the third in a three-part series about the financial side of things. In Part 1, talked about the systems we built for us to monitor network usage and for our users to buy and sell credit. In Part 2, we covered our operating expenses, and discussed some of the different considerations and models of how to bill users. In this part, I’m going to wrap things up by sharing some of the tricks and approaches that we can use to improve network performance and service for our users, and then providing some alternate funding models.

Local Services

In our network, a huge difference exists (in terms of network performance and congestion as well as cost) between traffic that stays local and traffic that goes out over the satellite link to connect to the Internet. Though our backhaul connection from our tower to the Internet is only 1 Mbps, the fronthaul connection from phones to the tower is much higher – we’ve seen speeds of 75 Mpbs for an individual user, and LTE offers theoretical speeds of up to 150 Mbps. Local communication, like sending a file from one person to another, doesn’t actually use the satellite link at all, and works even if we get cut off from the Internet. This brings up the question of “should we offer a local-only option?”

If we do decide to provide the option of local-only service to users, we also have to figure out what it should cost. There’s one argument that says we should charge them only for power, because that’s the really the only cost service they’re using. On the other hand, there are very good reasons for us to charge them more, and put that money towards the cost of the backhaul connection. Not only does everyone in the community benefit from Internet connectivity (non-paying users often request paying users to perform services for them), but we could potentially offer “lower priority” Internet connectivity when available. Remember – because the backhaul cost is fixed per month, every second we’re not using the Internet is a second that we paid for and simply didn’t use. Moreover, putting more money towards the backhaul could result in cheaper Internet access per user, which in turn incentivizes more users to join the network and creates one of those positive cycles I talked about in Part 2.

At the same time, we’re also actively working to improve what services are “local” by hosting them within the community on our server. If Gmail struggles behind a slow satellite connection, that’s a great reason for us to host a local email server. The exact same thing could be said for higher-bandwidth services, like streaming video with our own media server, or caching web content. I’ll save this discussion for another post, but we packaged CoLTE with many such services, including OpenStreetMaps and a chat server. Going further, improving Internet/Web performance at the network edge is one of my primary research interests, and I hope to pursue this topic at a painful level of detail before too long.

Limited Services

In addition to a local services option, we’ve also considered a “limited services” package that provides Internet access only to a select set of websites/services at a reduced (or free) cost. This package would likely be focused around calling and texting services (e.g. WhatsApp and Skype), but could also include other select websites such as Wikipedia.

The “limited services” idea stems primarily from the observation that many of the most socially useful services are also very low-bandwidth. For example, a standard-quality VoIP call requires only 20 kbps of bandwidth, potentially less under certain conditions, and WhatsApp text messages barely any bandwidth at all (Though these conclusions are intuitive and based largely on observed experience, we’re hoping to formally quantify this traffic soon!) These characteristics stand in stark contrast to other Internet services that provide a relatively low social benefit at a much larger cost of bandwidth, such as video streaming or online gaming. This dichotomy naturally led us to discussions around the idea of “we’re not making any money off of WhatsApp anyways, so why not just give it away for free?” Going even further, I’m almost certain that we could strongly prioritize VoIP traffic such as Skype and WhatsApp, and dramatically improve the social utility of our network, without our general Internet users even noticing the difference.

Thanks For Reading!

A fascinating theme that’s emerged in this series of posts is the intersection of technical engineering, socio-economic engineering, and what’s best for the individual community. One of the major takeaways of this entire series should be that the options are literally endless – especially since when the community is it’s own ISP, their interests are perfectly aligned. As we deploy additional networks in different areas, we’re hoping to build and research literally all of these topics, with the end goal being something as close to a set of “best practices” as possible. As always, thanks for reading, and please comment or reach out if you’ve got any thoughts or questions.