Helium co-founder and CEO Amir Haleem joins us once more to talk about the decisions that went into the hardware, software and design of the iconic Helium Hotspot. We cover security, hardware specs, resource consumption, and also take a look into a future where there are 3rd party hotspot manufacturers and DIY hotspots mining on the network.
Helium co-founder and CEO Amir Haleem joins us once more to talk about the decisions that went into the hardware, software and design of the iconic Helium Hotspot. We cover security, hardware specs, resource consumption, and also take a look into a future where there are 3rd party hotspot manufacturers and DIY hotspots mining on the network.
We also dive into Proof of Coverage, which is the method of verifying that real coverage is being provided by hotspots on the network, and has also been the rocket fuel behind the Helium Network's growth.
You'll hear some crazy ideas about old Proof of Coverage schemes, including sampling public radio, and where its design might go next, including coverage maps and witness sampling.
Fair warning, this is an in-depth technical discussion.
Learn more about Helium: https://www.helium.com/
Follow me: @rawrmaan
Follow Amir: @amirhaleem
Follow Helium: @helium
Intro song: Lakey Inspired - Arcade
Arman Dezfuli-Arjomandi (00:01:29): Amir, welcome back to the show.
Amir Haleem (00:01:31): Hey Arman. Thanks. Thanks again for having me.
Arman Dezfuli-Arjomandi (00:01:34): Let's talk about the Helium hotspot. What's its function, and what are some of the main considerations of its physical and software design? Because this is kind of unlike anything else that I've seen in the blockchain space, and it really looks more like a WiFi router than some specialty hardware for a brand new network.
Amir Haleem (00:01:53): Yes, so the Helium network is maybe a little bit different from some of the other networks out there. I mean, its purpose is to serve as a low power wireless network that sensors can use. And the underlying wireless technology we use is something called LoRaWAN, which is an open standard that's been around for a few years now, pioneered by this company called Semtech based in San Jose.
Amir Haleem (00:02:23): And there are a number of LoRaWAN gateways on the market already. So these are, you could think of them almost like WiFi access points, except they serve a LoRaWAN network instead of a WiFi network. And LoRaWAN is a very low power, low data rate kind of technology. So we're talking about data rates that are anywhere from 1,000 bits per second, to maybe 10, 20,000, if you're lucky. And so reminiscent of early internet modem speeds rather than anything broadband, and so you're not streaming video or voice or doing anything like that over this. This is a sensor network purely for telemetry kind of data like temperature readings and location pings and stuff like that. So there's a lot of hardware out here already. The LoRaWAN ecosystem is fairly large.
Amir Haleem (00:03:14): The challenge is that none of that stuff is particularly easy to use. The audience for that has typically been anywhere from telcos who look for enterprise grade stuff and hobbyists who are comfortable tinkering with command lines and using TFTP and SSH and all that kind of stuff. So that's on the IoT end, the radio end, the wireless end. On the blockchain side, we also wanted the mining parts of Helium to be as simple as possible. Like if you look at the cryptocurrency space, none of that stuff is again, particularly easy. If you wanted to set up a Bitcoin miner or an Ethereum miner, or really a miner of any kind, it is generally not very straightforward. There are some attempts to make this better with things like coin mine but they have their own ROI questions and everything else.
Amir Haleem (00:04:11): And so our goal was twofold with the hotspot, was like one, could we make an IoT LoRaWAN gateway that was easy to set up, and could we combine it with a crypto miner on our Helium network that was also easy to set up. So that really, we were broadening the audience away from just IoT enthusiasts and hobbyists or hardcore power users that know how to create Bitcoin addresses and wallets from scratch into a much broader audience that really could be anyone. And so we built the hotspot with that in mind and it's much less about novel hardware as much as it is user experience.
Amir Haleem (00:04:53): And so there's a relatively easy to use mobile app, and a lot of the work that we put into the hotspot was really just to try and make that user experience good. Like how do you configure a WiFi network? How does it connect to the right... How does it find peers on the block chain? How does it do everything basically, and get up and running easily. And that was really the thrust of all of our efforts in terms of the physical design and the software that runs inside it.
Arman Dezfuli-Arjomandi (00:05:22): Yeah. It kind of reminded me of setting up a Google WiFi router when I got my first hotspot and I've set up a few of them at this point, and it's really easy. I mean, it takes like five minutes. The thing itself is aesthetically pleasing, it can sit in someone's home without them worrying about guests coming up and saying, "what the hell is that weird thing sitting on your mantle" or "sitting at of your window". So I see why those were important considerations, if this is something that you want to have widespread physical deployment in people's homes. And I want to talk a little bit about some of the technical aspects of the device, because I think a lot of people have preconceptions about cryptocurrency mining, and that it's going to be this thing that you need some sort of industrial power connection to your local utility, and it's going to heat up the room and make your power bill go to $1,000 a month. But this isn't really like that at all.
Arman Dezfuli-Arjomandi (00:06:23): There's no proof of work mechanism or anything like that. And from my understanding, it uses about five Watts of power, which is not that different from, let's say, an LED light bulb. And I haven't noticed any difference in heat having a hotspot in the room. And even in terms of bandwidth consumption, it's not going to use up your whole internet connection. It actually uses about five kilobytes per second, which, I did the math, it's about 200 times less bandwidth than just streaming Netflix. So I think as a hotspot host, it's been very easy to make the decision to keep this thing in my home.
Arman Dezfuli-Arjomandi (00:07:01): I want to talk about some of the other hardware that's in there. So you built this hotspot on the Raspberry Pi, and I know that you've got some more hardware in there developed by RAK, and I'm not really a hardware guy, so can you explain what the process was choosing the components for the hotspot and was there any specialized hardware that you developed just for the hotspot?
Amir Haleem (00:07:24): Yeah. In the first iterations of the hotspot were actually a completely custom design that we had built. So we took a, they call them system on a chip, an SoC, that TI had built and it was like a dual core processor and it had a bunch of other processors inside it. And we built this software defined radio kind of architecture, and it was an entirely custom thing and it was very cool and there's like a ton of innovation in there. And I think we had built an SDR that worked really, really well at a very low price, which was very novel. But one of the challenges with that approach is that everything is very customized. If someone else were to want to build a hotspot or build a compatible device, they would somehow have to figure out how to replicate that.
Amir Haleem (00:08:17): And that is generally fairly expensive. If you wanted to manufacture on your own, even if we had open sourced everything, the one off manufacturing costs are very, very high. And so somewhere down the line, we decided that it would actually be better to adopt LoRa rather than this custom protocol, which I think I mentioned in the last podcast. And along with that came the option to take advantage of a lot of off the shelf hardware. And so we started talking to this company RAK, that's this Chinese company based in Shenzhen and they make these quite high quality, but low cost self contained LoRa gateway modules. And so a LoRa gateway is effectively an eight channel receiver. So it can listen to eight different LoRa channels at the same time.
Amir Haleem (00:09:11): And then that gateway module has to talk to some kind of process or some sort of computer. And the easiest thing that we could think of using was a Pi. They're relatively cheap. They're available everywhere. There's good support. Most of the bugs have been figured out in terms of all the different interfaces like WiFi and Bluetooth. There's good support for Buildroot, which was the custom Linux kernel in-build system that we've used. And so that was the decision we made, was to head down that path rather than this proprietary or custom built hardware path. And so the hotspot is really like three major components, so there's a Pi, there's a thing that we call internally the side table, which is the board that the RAK LoRa gateway module sits on.
Amir Haleem (00:10:04): And that side table also has a bunch of other stuff in there. There's for example, that's where the LED light sits that you see on the outside of the case. It's also where the button controller sits. So there's a button on the side of the hotspot that activates Bluetooth. That's where that sits. It's also the power supply, we didn't want to use USB or micro USB power. We thought it was too flimsy and so we have a proper barrel type power connector on the back of the hotspot. And then maybe most importantly in the crypto side, there's also a secure crypto element in there, which is where the private key for the hotspot lives. And so that's really what it does, is it sits there, it interfaces between the Pi and the LoRa gateway module, and then it provides us with these ancillary things that we thought were important in order for the hotspot to be easy to use and secure.
Amir Haleem (00:11:03): And there wasn't really anything out on the market that we could find that did exactly that. There's also GPS on there, I forgot to mention--there's GPS on the side table, too. We would expect that to change over time. Like now that I think we've shown a little bit of a blueprint of what you could build. We've already seen a couple of gateway manufacturers emulate this a little bit. They're now starting to include a button and a light and a better power supply. And so I think this is all positive because those are all things that you want in order to make a good consumer experience out of this thing. And I think Helium is really the first time I think that anyone has sold significant volume of LoRa gateways into the consumer space.
Amir Haleem (00:11:47): And I think that has caused everyone in that ecosystem to at least pay attention. It's like, there's a different audience for LoRa than they thought. And I think most of those guys, and perhaps some of them, like we've seen some internal emails that make us giggle, some still think this is a weird scam kind of thing because of the crypto currency side. But for the most part, everyone in the ecosystem is now keen to either work with us or build compatible stuff for the network. And we've shown them a way to do that and at least what we, or our customers expect the ecosystem or the user experience to look like.
Arman Dezfuli-Arjomandi (00:12:26): Well, hopefully a rising tide lifts all boats in that respect. And I definitely have more questions for you on the manufacturing side, but you mentioned security. So how secure is this device? I think this is a concern that a lot of people have with their hosts. Will this be another vulnerable device on someone's network with tons of open ports and the ability to talk with other devices? Or is it fairly locked down?
Amir Haleem (00:12:47): I mean, you never want to say that it's like super secure because that always seems to end badly, but I think we did all the obvious things quite well. I mean, most of the IoT vulnerabilities that you see, like Mirai, for example, which was this huge botnet thing are nearly always driven the same way, which is that you've got a device sitting on your network, whether it's an IoT gateway or a printer or a fridge or whatever the hell it is, and that has an SSH server running and the credentials for that SSH server or Telnet are some combination of dictionary words. So the login is root and then the password is whatever.
Amir Haleem (00:13:35): And generally, the way that those hacks have worked is that they identify what that is and then they just go and find every single device that shares that combination and then they have access to it and they can do bad things. So in the hotspot, there is only one port open, which is the TCP port for the peer to peer network. You don't have to have it open. The way our peer to peer system works, it can discover other peers without a direct connection, it's just harder and slower. So I think we have port 44158 in TCP open. But I would always recommend--and there's no other ports open. So there's an SSH client running, but there's no SSH server, everything is an outbound dialing kind of thing. The crypto element is the private key is secure on this crypto element.
Amir Haleem (00:14:31): So you can't actually ever read the key, you don't have any idea what it is. You can just have the crypto chips sign things for you, which is how sort of the assert location and add gateway transactions work, is that you just ask the chip to sign these things and it signs them and returns the signed value, but you still don't know what the actual key is. And so it's a cool chip. It actually was originally derived from printer cartridges, believe it or not, where they were trying to do DRM on cartridges so that the printer knew that it was an authentic cartridge. It's funny how this stuff works, but those chips are now widely adopted, and I think we were some of the first to start using them back in probably 2013.
Amir Haleem (00:15:13): But that's where the security lives, is that we tried to lock down every port that that is vulnerable, which is basically all of them. I would still recommend to anyone that they should put devices like the hotspot and really anything in a DMZ firewalled zone on their own network. We think it's secure. I'm sure there's some vulnerability, but we've done a bunch of external pen testing and auditing and so we feel good about it, but nothing is ever perfect. And the best thing that I think you can do for yourself is put it on an external--in an externally gated environment where it doesn't have access to your internal network. But as far as compared to other devices that we've seen, I think we did a good job in securing this and making it easy to use at the same time.
Arman Dezfuli-Arjomandi (00:16:00): How is data that's transmitted through the hotspot secured? What kind of data is flowing through this thing? Can you talk very briefly about are people able to just send random packets to the internet or is there some more limited fashion that data is being transmitted here?
Amir Haleem (00:16:13): I mean, there's two sets of data that flows through the the hotspots. So there's peer to peer traffic, which is your hotspot--we call this gossiping--but basically sending blocks of transactions that it hears over the network to other hotspots. It's how a blockchain peer to peer network typically works, is that this stuff is broadcast across the network and it's shared around by all the participants in the network until eventually everyone has it. So everyone gets the same blocks and everyone gets the same transactions because they're all being gossiped around this network. And for that we use and implementation of libp2p, which was designed by Protocol Labs who implemented IPFS and Filecoin.
Amir Haleem (00:17:00): We wrote our own implementation in Erlang that added a few things that we thought were necessary given the consumer nature of our products, so things like NAT traversal, so you can navigate through all the different firewall setups that people have at homes to UPNP and PMP and all these other PMP protocols that everyone has implemented on their different WiFi routers. None of the other P2P implementations at the time had that, maybe that's changed now, but we thought we needed that so that someone could just stick a hotspot in their house and it would automatically detect what kind of NAT situation they had and then traverse it somehow.
Amir Haleem (00:17:41): And there's some standard protocols for doing that, that are called STUN, TURN an ICE. And so we had to implement those in P2P in order for this to be viable. And other blockchain projects probably don't have this problem where their miners or the peer to peer network is assumed to be running in data centers or warehouses or something like that. But we knew that a lot of these would be on consumer home connections, and so we had to figure that out. And so that traffic is one type of traffic that flows over the network. And then the other is the wireless traffic that's being transmitted by sensors. And that's not IP traffic. Those are not TCP/IP packets, they are LoRaWAN packets, and your hotspot acts as a bridge between LoRaWAN on one side and then the internet on the other.
Amir Haleem (00:18:31): So it hears these LoRaWAN packets using the gateway module that I mentioned, they get encapsulated into adjacent object and then they get delivered to whichever server they're supposed to be delivered to on the other end. And those packets are encrypted with AES128. LoRaWAN security protocol is fairly well documented, we didn't do anything different there. And so that's super low bandwidth stuff. No one is going to be sending... no one even can really send anything malicious over LoRaWAN. Although I probably shouldn't say that, but no one's streaming video or something over LoRaWAN it's not really practical.
Arman Dezfuli-Arjomandi (00:19:11): Yeah. You're limited to pretty low data rates there. So I guess the level of malicious... I mean, I guess you could send some really naughty texts or something, but there's a pretty limited range of things that you could do.
Amir Haleem (00:19:22): Yeah. And even then they're encrypted and they go to the server end. So it's not that exciting, it's being delivered to wherever it is the sensor owner decides they should be delivered to. And your hotspot doesn't really know a whole lot about it.
Arman Dezfuli-Arjomandi (00:19:37): Right. So it's not high risk, like hosting a Tor node or something where there's just tons of traffic going to the clearnet or whatever. It's just basically encrypted packets going to these routers and no one can even decode what's in them, and they're small.
Amir Haleem (00:19:52): Yeah. I mean, and ideally Tor would be the same way. I'm a big fan of it. And ideally, no one would really know a lot about it, but it sounds like there are at least some concerns about how many nodes are operated by like the DOD or whatever that seems to compromise that system. But similar idea, the stuff is encrypted, the host doesn't really know anything about where it's going or who owns it. But in the case of LoRaWAN, you're limited to very, very small payloads. The maximum payload size is 255 bytes. And then at the longer range modes, I think you're limited to like 11 bytes, or maybe it's a little bit more than that, but it's close to that. So there's not a whole lot of damage you're doing sending an 11 byte non TCP packet over the air.
Arman Dezfuli-Arjomandi (00:20:38): So I want to talk about antennas for a sec. The hotspot has a replaceable antenna and some people in the community have gotten pretty ambitious putting huge antennas on the roofs, even having really tall indoor antennas with magnetic base mounts. But the builtin antenna actually has pretty solid performance in my own testing, even easily beating some other antennas with higher dBi ratings. So what made you choose the stock antenna? What's so special about it?
Amir Haleem (00:21:05): So the stock antenna is from this company called Link Technologies. And we had started talking to them after noticing that like pretty much every high quality radio dev kit that we were aware of, including the LoRa kits and stuff from TI and others came with these antennas. So they're black with little yellow bands on them for the US spec antenna. And they're good. They are quite consistent, they perform well, they don't have the highest gain. So the gain is a measure of how much they can amplify what they hear, I suppose, one way of describing it. But they're simple to use. You can't really screw it up, they're a dipole type of antenna, so they don't need external ground or anything like that. And you literally just screw it in and make sure it's pointed up and you're good to go.
Amir Haleem (00:21:59): As you mentioned, some people in the community have started to get crazy with the different antenna choices, which I think is great. But there's a lot of unknowns there. Like picking antennas is not a straightforward task, and a lot of it depends on where you mount it and how and there's a little bit of concern there that people are just doing things that they don't exactly understand, and sometimes it works sometimes-
Arman Dezfuli-Arjomandi (00:22:25): I'm guilty of that.
Amir Haleem (00:22:26): ... sometimes it doesn't. And no one really understands antennas other than 12 people in the world or something. And so there's a little bit of that and we didn't really want to get into that too much, we wanted it to be as easy as possible to set up and use. But yeah, I mean, you've just got to be careful with some of this stuff. I mean, I see a lot of people using highly directional, like Yagi antennas, which you point in a certain direction and have extremely high gain and those look like a TV antenna. And those are okay, they have extremely long range, but they only go in one direction, so your hotspot wouldn't be hearing any traffic coming from the other direction.
Amir Haleem (00:23:09): You're also probably violating FCC rules with an antenna like that because there is a maximum amount of power that can be radiated from these kinds of things. And so there's a lot to think about there, and so sometimes I see people saying like, yeah, you should use this antenna. And maybe in some situations, that's going to be true and in others, that's going to be a bad idea because you'll only be hearing things from one direction and the FCC probably wouldn't be happy with if they found you.
Arman Dezfuli-Arjomandi (00:23:36): Well, not to mention, it's not really benefiting the network, right? If you're only providing coverage in some straight line.
Amir Haleem (00:23:44): Yeah. I mean, it can be. There's certain settings if you're on, I don't know, like the outskirts of San Francisco and you have them pointed into the city, that's probably not terrible, but still it's probably just, it's questionable because you're still in the FCC danger zone. And it's probably better to have omnidirectional antennas when you can, that are similar to the one that the hotspot comes with. Really the best thing that I think you can do without getting crazy is sort of figure out how to position the antenna higher and in more space. I kind of like to think of antennas like a shotgun, the more space that you have away from the shot gun, the further that it spreads.
Amir Haleem (00:24:31): But if you had like a wall in front of the shotgun, then it doesn't get to spread very far. And so putting the antenna up high in open space is always the best thing to do regardless of what antenna it is. And so we've had good success internally with just buying magnetic extension bases for the antenna and just running them outside or putting them out on a balcony or putting them on a roof or whatever, but using the same antenna. And that works really, really well. So you don't have to get crazy. You don't have to buy one of those super high gain, big antennas, but generally if you know what you're doing and you install it well, it's only a benefit to have a better, bigger omni directional antenna.
Arman Dezfuli-Arjomandi (00:25:14): Spoken like a true FPS gamer with the shotgun analogy.
Amir Haleem (00:25:20): Yeah. It's just how I think about it for whatever reason, it made sense to me that way.
Arman Dezfuli-Arjomandi (00:25:26): So speaking of customizations, it's currently possible to make a DIY hotspot with your own hardware and the open source Helium miner software. So like you can get one running on a Raspberry Pi, you can transmit packets through it, but they can't participate in Proof of Coverage or earn token rewards yet. So how are you thinking about bringing these untrusted devices onto the network where Helium doesn't necessarily control the software?
Amir Haleem (00:25:53): Yeah, so our desire is that we are just one of many vendors in the Helium ecosystem. We built the hotspot as we talked about for a set of reasons, but we want any LoRa gateway to be able to participate in mining as long as the network can prove that they're actually creating useful coverage. And so the hard part there is that you need to somehow protect against these virtual machine actors, people who actually don't have a LoRa gateway or any LoRa hardware and are just pretending to be generating packets or being able to hear packets. And there's no easy way of verifying whether something was really sent wirelessly or not. By the time it gets to the network, it is just a JSON object. And you could have generated the contents of that object anywhere.
Amir Haleem (00:26:49): So the way we do it today is a little bit of a hack where we know for sure all the actors involved in the network are actually running the right hardware because they came from Helium Inc and they have the secure crypto chip and we know what those addresses are. And so the step that we're trying to get implemented now is like, how do we make it possible for absolutely any kind of hardware to join the network and do it in a relatively safe way? And so there was a HIP out there which is a, what we call a Helium Improvement Proposal, very much copied from Bitcoin, that describes a way that this could work where there's a leveling scheme where gateways join the network with the lowest possible level and level up as a result of interacting with other gateways or hotspots that are relatively trusted. I think it's an okay system. It's certainly a lot better than what we have in terms of how secure it will be in this context.
Amir Haleem (00:27:53): And the intention is sort of, as we said, we don't want to be the only hardware vendor here at all, that's not the intention. We want to be able to open this up to absolutely any hardware that's compatible with the LoRaWAN protocol. And that's what we're working towards diligently. And there's a lot of good proposals coming out of the community as well to figure out better ways to continually do this. And I think it's just going to be an evolution. There are not any other blockchain networks that I'm aware of that have this physical dependency on things, and it just adds like another complication. We're trying to prove that you're creating network coverage and that on its own is just hard and difficult. And there's a lot to be figured out and a lot of ways I think we can continue to improve this over time, and we hope to be not the only contributor in terms of how future upgrades and improvements get rolled out.
Arman Dezfuli-Arjomandi (00:28:50): Yeah. And so for the next batch of hotspots, are you going to be producing those yourselves or is there a third party manufacturer that you're working with to build a new model?
Amir Haleem (00:28:58): Well, we're certainly working with as many LoRa manufacturers as we can to just encourage them to build stuff that is out of the box compatible with Helium. And that doesn't really take much because now that we've switched to supporting LoRaWAN several months ago, any LoRaWAN compatible gateway is in theory usable on the network. And that the question is just how to make the user experience good enough that it at least resembles what our hotspot did or does. And so the question is mostly around UX rather than a technological challenge or, maybe they're related. But for hobbyists and enthusiasts, it's absolutely possible once this level scheme rolls out--and we hope that that will happen by the end of summer--to take any LoRa gateway that you have, whether it's a $50 Chinese version or a $5,000 telco grade version, which exists, and connect it to the network and participate in mining. That's the goal.
Arman Dezfuli-Arjomandi (00:30:05): Got you. So when you say mining, there's a couple of things here. There's Proof of Coverage, which I want to take a quick dive into, and there is data transfer. And as of July 27th, hotspots will be receiving rewards for both of these things. And so data transfer, you can kind of trust any hotspot to do that because it's just data coming through the network, it doesn't really matter where it's happening and the data has to be paid for some way or another. So there's really no trust issue there, but Proof of Coverage is really just this mechanism where hotspots are rewarded for existing. So what are some of the key insights behind Proof of Coverage? Because this has kind of been the rocket fuel behind the growth of the Helium network.
Amir Haleem (00:30:52): Honestly, we were a little bit inspired by Filecoin here. Perhaps not directly, but certainly indirectly because we had an interest in doing something like this and we didn't really have a great idea about how we would do it. Obviously, something like proof of work or even proof of stake is not particularly useful in this context, because what we really wanted to do is encourage people to like build a network, build a physical coverage network. And so some of the things that you wanted to incentivize in order to do that was obviously first, have the right kind of hardware. Are you running a LoRa gateway or not?
Amir Haleem (00:31:28): And then also to be spread out enough. Like actually position these things such that you are being rewarded if it can be proved that you are creating a broad coverage area or participating in a broad coverage area. And so Filecoin was like I think, at least for me, the first time that I had seen a practical application of a proof that was related to what the network was trying to do. So Filecoin is a file storage network that's a decentralized S3 or something like that. And their proofs were all about could you actually prove that you were storing the file over time and that you still had it and that you could make it available if someone needed it back. And Proof of Coverage was in an attempt along the same lines, really, which is that could we somehow prove that you were creating network coverage and reward you in proportion to how much network coverage you're creating or how much of it could be proved to be yours.
Amir Haleem (00:32:27): And that was the original insight behind it and the original idea, because at the start of these networks, and I get that Bitcoin was really the pioneer here, is that you have to figure out a way to reward people for both building and securing the network in the absence of it really being heavily used. At the start there's no traffic, there's no like hundreds of millions of devices on the network. There's not even one device on the network at the start. And so you have to figure out a way to incentivize people to start doing that. How did you start from scratch and get this network effect rolling when there's not a bit at the start, it's the chicken and egg thing. And so I think crypto networks in general are very smart about this.
Amir Haleem (00:33:14): This is the most interesting thing to me about the crypto network effect is that you get to incentivize these early participants and disproportionately so. The earliest participants generate the most rewards for themselves, like the earliest Bitcoin miners are rich now. And that's an interesting characteristic that I think some people have exploited very well or taken advantage of or built products around very well, and others have just squandered due to selfish token distributions and just other..what I consider really shortsighted decision making. But that's what I think is really interesting about crypto networks and Proof of Coverage is that, right? Like how do we figure out a way to prove that you're running the right hardware and actually it can be seen and heard.
Amir Haleem (00:34:05): And so it's nothing to do with electricity, it's not a proof of work scheme. And proof of stake, really didn't seem to really help us there, that didn't add any particular value. So when we came up with a scheme where hotspots generate challenges, which are, you can almost... it's funny that you mentioned Tor because the construction of a Proof of Coverage packet is very much like Tor where it's a layered onion. And each layer of the onion is only decryptable by specific hotspots. And so the way you can think of a Proof of Coverage challenge is that there's a Challenger, which is a hotspot that is nowhere near... it doesn't have to be anywhere near the group of hotspots being challenged. So that can be in Florida, and what it does is create a challenge packet by looking at the blockchain and saying like, okay, these are seven hotspots in San Francisco that claim to be in San Francisco.
Amir Haleem (00:35:06): If I construct a onion packet that is decryptable only by those hotspots in a specific order, and I can do that because I know what the public keys of those hotspots are. And so one of the characteristics of public key cryptography is that you can actually create something that is only decryptable by that private key by owning the public key, which is awesome. And so you can create this layered onion packet that is only decryptable by those seven hotspots in a very specific order that they're supposed to be able to hear the packet.
Arman Dezfuli-Arjomandi (00:35:41): Right.
Amir Haleem (00:35:42): And so if you laid out this path with like one, two, three, four, five, six, seven hotspots: Layer one is encrypted such that only hotspot one can understand it and layer two, et cetera, et cetera. And then it delivers that packet to the first hotspot over the peer to peer network. So that first step happens over the internet and then subsequently hotspot one gets the packet and then it broadcasts it over RF. So using LoRa, it sends the packet over the air and if hotspot two is where it's supposed to be, then hotspot two will actually hear this packet and it will be able to decrypt its layer because it's encrypted only for hotspot two. And it delivers a receipt back to the Challenger over the peer to peer network. Thus it says I got this, I was able to decrypt it, and here's my proof of that and I'm sending it back to you.
Amir Haleem (00:36:30): And that keeps happening and then it broadcasts it again and then hotspot three should hear it and it should also be able to decrypt and deliver the receipt back. And then at the end of that, either after a timeout or after hotspot seven has delivered its receipt, the Challenger submits its pile of receipts to the network and then everyone gets rewarded, basically if that worked out. And everyone is incentivized to do the right thing there, the Challenger gets an HNT reward for constructing the challenge and delivering the receipts and the Challengees in the past get rewarded for being able to verify that they were able to receive this packet. So the neat thing about this construction is that none of the hotspots in the chain have any idea who's next, and so it's very difficult to trick this process in terms of trying to construct a fake packet or delivering fake receipts because there's no way to really know who was supposed to be next.
Amir Haleem (00:37:25): The Challenger doesn't reveal that at any point until that challenge has already been recorded in the chain or been mined. And so that's what we were going for there, is like, how did you create a thing where someone running a fake virtual machine network would have a hard time faking all of the activity going on in there because there would be no way for them to know in this random, but deterministic way, what was supposed to be happening.
Arman Dezfuli-Arjomandi (00:37:52): So in Proof of Coverage, you have these multiple elements. You have Challengers, Challengees and Witnesses, and I'm not going to get into it here due to limited time constraints, but I would encourage anyone to go read the Helium developer docs because they explain everything in a really clear way and what the relative value of each part of the equation is. But what are the requirements to participate in Proof if Coverage? Does a hotspot needs to be witnessed by nearby hotspots or issue challenges in order to be challenged?
Amir Haleem (00:38:20): Today no. But well, kind of. So the way it works is that when you first join the network, your hotspot is in what we call beacon mode. No other hotspot has heard it, it's new. And it sends out a beacon, which is, you could think of it as a single layer PoC packet. And it sends it out in the air and any other hotspots that are nearby will what we call "witness" that packet. It will hear that packet, and then every hotspot uses that witness data to inform which hotspots it should be able to hear in subsequent challenges. And so you've got this graph of graphs being constructed throughout of like hotspots who have witnessed other hotspots.
Amir Haleem (00:39:17): And so the docs do a probably a better job then I could to explain how this effectively get started. But yeah, when you first start, you're invisible until other hotspots hear you. And if no other hotspots can hear you, then you never get included in a challenge path. If you're a loner in the middle of nowhere with no other hotspots around you, the only function that you can perform is creating challenges. So creating those encrypted layers and delivering them to other hotspots until another hotspot is added to your area. And so being on your own is not useless, you're still adding some value to the network, but because it can't be verified that you're actually doing anything, you can't be verified that you're there because no other hotspots can hear you, you get minimal rewards relative to what you would get if the were other hotspots in the area.
Amir Haleem (00:40:07): And so typically we see hotspot purchasers buying two or three at a time and having one at their house and one at a neighbor's house and one at a friend's house or whatever, so that they're actually creating this witness graph rather than just being on their own.
Arman Dezfuli-Arjomandi (00:40:23): Yeah. I've noticed that the hotspots that earn the most tokens currently are the ones that have plenty of other hotspots to communicate with, as you just said. So how can people optimize their setups for Proof of Coverage earnings and to maximize value for the network? And what's the incentive for someone to cover a completely new area without buying two or three hotspots? Or is there any?
Amir Haleem (00:40:46): One of the things that I think is neat is that when you're in a new area, it is highly likely that when you are challenged, that your hotspot participates in a challenge. And by that, I mean if you're in San Francisco where there's like 300 plus, maybe even 400 hotspots, the probability of you being included in a challenge in that area is lower perhaps than if you're just in an area on your own, assuming that there are other hotspots around you in that area. So generally, it's a good thing to create a new coverage area. So if you go to a new region and you take two or three, four, or five hotspots with you, that generally seems to be a good thing, like the system rewards you well for creating consistent coverage in that area.
Amir Haleem (00:41:33): But generally, yeah, I mean, you want to have as many hotspots around you as possible to verify that you're actually doing what you're supposed to be doing. And the only way that Proof of Coverage can know that is by this independent verification from other hotspots. And so I wish there was a way that a lone hotspot could prove that it was really doing the right thing. And we had all sorts of crazy ideas, like maybe it could sample some FM radio or some AM radio in the air, and is there a way to verify that? And you really need some system of being able to verify. It's one thing to just do something, but it needs to be verifiable and deterministic in some way that the rest of the network can look at it and say, yeah, okay, that did what it was supposed to do.
Amir Haleem (00:42:21): And in Proof of Coverage, anytime one of those receipts is submitted by a Challenger, every hotspot on the network verifies that it was the right thing to have done. Is that actually the correct path that should have been created and are those the correct receipts? And if they aren't, then the transactions get rejected and they never get mined and no one gets rewarded. And so you have to have some way of it being deterministic and verifiable. And so sampling something like AM, or FM radio in the air is a neat idea, we just don't know how to verify it in some way. Like how does someone look at that spectrum that was sampled and say, hey, that's actually the exactly correct... that's what that should look like if you were in that area.
Amir Haleem (00:43:08): And so again, I think this is like ongoing research and work that needs to be done. And there's all sorts of interesting ideas and ways that we could imagine this working, but solving the lone hotspot problem, I think is very, very difficult for that reason because there's no other external way to verify what you're doing. And so really, the best way is to accompany that lone hotspot with others.
Arman Dezfuli-Arjomandi (00:43:34): Yeah. There's a unique set of challenges with building for blockchain in this kind of distributed closed loop system, where without having any external oracles, which are still by and large, an unsolved problem, you can't really know what's going on outside of the system or bring any external verifications into it. But that being said, you've come up with a lot of iterations of Proof of Coverage.
Arman Dezfuli-Arjomandi (00:43:58): If you... I recommend to anyone who's interested in the evolution to go check out the engineering blog at engineering.helium.com, because it's very transparent. They give you a summary of all the software updates, like in a technical way, but also in a human readable way and explaining the reasoning behind each update, which I think is really cool. And you guys have had a lot of iterations of Proof of Coverage. I think I joined the network when it was around V2 or V3 or something. And now we're up to, I don't know, V7, something like that.
Amir Haleem (00:44:29): Yeah. I think we took a different approach to this than a lot of other projects. I don't know if it was right or wrong, honestly, if I'm just being honest, I think what we've seen other projects do is just not launch until it's perfect. And I think that's a way to do it, and you could certainly argue that that's the correct way to do it. And I would believe most of those arguments, but the other way that we've taken to do it is let's get a good effort at it, a very good effort at it that that does what we want to do. And then learn from the feedback that we get both from the community and from a data collection point of view about what happens. Because what we've seen, especially with these consumer related projects or products is that you don't really expect the feedback that you get.
Amir Haleem (00:45:24): At least I never have in anything that I've ever worked on. And getting something out there is valuable in that regard. Is that you get to focus on the things that are the highest value problems versus the things that you think are the highest value problems. And so I don't regret that decision at all, but one of the implications that it has is that Proof of Coverage is certainly not perfect. There are issues with it, there are challenges and problems and bugs, and it's a never ending challenge. I think Anatoly from Solana when I was on the podcast that they have described blockchains as like an infinite problem, and that's really what this is. And adding this physical proof element to it just makes it even more infinite, if there is such a thing.
Amir Haleem (00:46:12): And so I think waiting and just trying to perfect it, we arguably would never have launched because it would always be a thing and it would always be an improvement and always be a fix. And so I think it was more important to prove that this mechanism of building wireless networks is valid, that this incentive scheme works and that this economic model has value. And to improve Proof of Coverage over time, also bearing in mind that Proof of Coverage becomes less and less important a part of the network over time. Like we have this diminishing reward schedule where over time data transfer should be the predominant way that hotspot hosts earn HNT on the network, and PoC is really as a starter. It's like how you kickstart the network in the same way that Bitcoin mining rewards are going to zero.
Amir Haleem (00:47:07): And it's intended that miners will earn all their fees from transactions, it remains to be seen how that's going to go. But our system is similar in that way, is that over time we expect all of the rewards going to hotspot hosts to really come from data flowing over the network. So this proliferation of sensors and applications that we hope to see on this network should be how hotspot hosts earn HNT rather than PoC, which is a solution to the problem that you don't have activity at the start.
Arman Dezfuli-Arjomandi (00:47:40): Big shout out to Anatoly and the Solana team for your mention of No Sharding. I recommend anyone to go and check out that podcast. It's a huge inspiration for this one. So I know we only have a couple of minutes left here. So very briefly, what's the future of Proof of Coverage? What's in the pipeline?
Amir Haleem (00:47:57): Well, there's the HIP that we talked about there, which proposes this level scheme. There are some other internal discussions that we've had that we want to open up to the larger groups that relate to whether location is an important consideration for Proof of Coverage. Like today a lot of it is location-based, but it's probable that that doesn't actually matter very much. And there's certainly a rationale there that exists that you might not need to care about where a hotspot actually is in order to be able to prove whether it's creating coverage or not.
Amir Haleem (00:48:35): And so that's a very interesting concept and a very interesting notion because proving location of something over the air is quite challenging. Not only do you need very accurate timestamping of data, which we have in the V1.1 and above hotspots, it can get us down to like under 10 nanoseconds of timing accuracy, but you also have multipath packets and RF stuff, which is when data transmitted or symbols transmitted over the air get reflected off things.
Amir Haleem (00:49:11): So if you imagine you were trying to time how long something took to get from point A to point B, it gets hard when it may have bounced off of 12 buildings before it had gone from A to B. You no longer have any idea how long it should have taken. And so in cities we're no longer very confident, and we spent a lot of time with Semtech on this, we're no longer very confident that regardless of the timing accuracy, that you can get anywhere near the precision that we would've wanted. And so for some applications, that's going to be fine, but for locating something within a few feet, or even tens of feet in somewhere like San Francisco, we just don't believe it's possible at scale.
Amir Haleem (00:49:54): And so that changed a lot of the way that we thought about Proof of Coverage, because the location was always a very big part of it. But again, this is one of the things that you learn from actually deploying a network, rather than just thinking about it, is that you get to experience a lot of this in the wild, and there's only so much testing that you can do that resembles the real world. You can get very good location accuracy using this time of flight measurement in a controlled setting. Like if you imagine having a bunch of hotspots in a circle with a thing in the middle, you can almost perfectly know where that is. But you start adding reflections and buildings and inaccurate GPS timing, and all sorts of other stuff into the equation and it just starts to get harder.
Amir Haleem (00:50:38): So a lot of the thinking around that Proof of Coverage future might involve discarding location as a first class citizen of the system, which to me would be very interesting because of revealing everyone's location is the part of the system that I'd like the least. And so I would love for that to come to fruition and for that discussion to go further, but it's again, very early stages. And we hope to involve the community significantly more in how we form some of the stuff. And we would love to bring more engineering power into the equation.
Arman Dezfuli-Arjomandi (00:51:13): So that's really an interesting idea of ditching location. Can you explain a little bit the idea of why it might not matter where hotspot is?
Amir Haleem (00:51:24): Again, this is super early type of theorizing. Not a whole lot of work has gone into this idea, but as I've thought about, or as collectively, I think we realized that you can prove that something is creating coverage without necessarily knowing where it is. And this is an interesting idea. It changes a lot of the ways that we would think about Proof of Coverage, and I don't know if it's even feasible or viable, by the way. This is all-
Arman Dezfuli-Arjomandi (00:51:52): Just conjecture at this point.
Amir Haleem (00:51:53): Yeah. All very conjecture-y, but it's interesting. Do you actually care where the thing is, as long as you can prove that it can hear RF? That's really the discussion.
Arman Dezfuli-Arjomandi (00:52:05): Right.
Amir Haleem (00:52:06): Do you actually care that it's in my house, as long as a sensor in the area can send data and have it delivered to the internet? Is it actually relevant to know where the hotspot is? And that's again a super interesting idea, and as you dig into it a little bit more, it becomes maybe more obvious why that may be feasible. But it would change the way that PoC certainly works today, where a lot of it is based on where something is, you know the entire structure of that system is based on that. And so yeah, it just depends. And again, there's a lot of stuff... It's a very fluidly moving landscape. At the start of this project, we thought that GPS-less location was highly valuable, and after dealing with customers and deployments for the last like three, three plus years, that no longer seems to be true.
Amir Haleem (00:53:05): GPS is low enough power and now there are chips that incorporate GPS and WiFi sniffing and cellular base station sniffing into a single chip that give you like additional location insights, and so for most users of the networks or customers that don't care about the crypto side at all, they just want to build a product like a Tile or something, those are much more practical ways of solving that problem than trying to do the time of flight thing where you need really a massive amount of density across a whole country in order to make that work, because with just one hotspot, hearing you is no longer sufficient, you need like four or five, six, seven, or eight to really get good location accuracy without using GPS.
Amir Haleem (00:53:53): And so again, that's like all of this stuff that we've learned just from putting this out there and dealing with potential customers, like really helps inform where we should be spending our time. And so that's become one of the things. It's like, okay, well, how important is location in the equation? And so there's some discussion around that. It may be that everyone realizes location is really important and that still needs to be a first class citizen. Today we use it as a type of Sybil resistance in a way, like you can't be in two places at the same time, and that is part of what helps Proof of Coverage work.
Amir Haleem (00:54:30): And so there's a lot to figure out there. And again, it's very early and maybe even stuff that we shouldn't be talking about too much because it's miles away from any practical implementation. But the point I think is more that there is more work to be done on PoC, and it needs a lot of thought and a lot of ideas, and there are a constant, never ending stream of things that I think we could do to make it better and that everyone can contribute to, to make it better. But it hasn't been bad by any means. And I think as you said, it's been a great fuel to build the network and get it deployed even though we know it's not perfectly perfect.
Arman Dezfuli-Arjomandi (00:55:12): Yeah. I think the idea of the network not caring where hotspots are located is already baked in, right? I mean, you don't give any priority to any urban area or any location, really. It doesn't matter if you're deploying in New York city or deploying in rural Virginia, you're still going to get the same reward regardless. And so in a certain sense, the only reason that locations currently exist is just for people to have a general idea of where hotspots are, but in terms of the actual functionality of the network, it doesn't really provide any utility. And I know you guys are embarking on some coverage mapping efforts using third party devices, LoRaWAN devices that have GPS built in, and that's a pilot program at this point, but are you thinking of creating coverage maps in a way that would eventually end up on chain or is this just an exploratory initiative?
Amir Haleem (00:56:06): Yeah, I don't know, honestly. There is... so one of our engineers put together something that I've wanted to do for the longest time. So I'm so glad that he did this, which was to take all of the witness data that's on chain and turn it into a heat map of coverage. And I don't know if this has been shared yet, but we have it. It's awesome because this is actually real coverage information that is autonomously generated through the Proof of Coverage process. And so I don't know if he's shared that with anyone, but it's really cool to look at because rather than just the theoretical modeling that you can do using tools like cloud RF and everything else, this is actual like, okay, one hotspot transmitted and another hotspot witnessed, and that is what informs the heat map that was built there.
Amir Haleem (00:56:59): So I think that's one way to do it, which is great and probably the most scalable way, because it doesn't involve needing to drive around or whatever. But I also really liked the way that TTN did this. So The Things Network has a project called the TTN mapper and it's this open source initiative where anyone can build a GPS device that is compatible with the mapper protocol and just wander around like transmitting data into the mapper that then the mapper uses to create a coverage map. I love that idea. I think that's a fantastic initiative. They did a really nice job on that. And so I think we want to do something similar where people can build a compatible device of some kind and walk around and build a coverage map in their area. Whether that should be on chain or not is debatable, I would guess no.
Amir Haleem (00:57:56): The witness data already is on chain and serves that function a little bit, like you can build the heat map the same way that one of our engineers did. So I would argue you don't need more data, you want to be careful about the ledger and how much stuff you're storing on there. But there's all sorts of stuff that people can do with or without our assistance, all of that lives on chain and you just need the time and effort to go build it. But yeah, so curious to see what happens there. I mean, the TTN map is an open source thing. I'm sure someone will fork it and do a version on Helium one of these days. But I really like those open initiatives around coverage mapping.
Arman Dezfuli-Arjomandi (00:58:42): Yeah. I love the idea too. And I definitely fantasize about how can you give a useful device to like say, Uber drivers, that you can give them for free or somehow subsidize the device and just pass out devices to people who might be moving across vast amounts of terrain to create this crowdsourced coverage map. And I know that there are some apps out there that are doing something similar. I think XYO is one of them. I don't know too much about it, that might not be accurate, But it seems like this is more of a smartphone based location mining thing. That's really interesting. So I'd love to see if something like that could be combined with the Helium network. Because I think robust coverage maps can be really useful for anyone who's considering deploying in an area.
Amir Haleem (00:59:27): Yeah. I mean, eventually they become a must, like you need to have some decent sense of what the coverage looks like. And you can look at the hotspot map and get a sense of it, but sometimes it's non-obvious. Like in one direction, the hotspot in my house creates about half a mile of coverage and in the other direction it creates about eight miles of coverage.
Arman Dezfuli-Arjomandi (00:59:47): Right.
Amir Haleem (00:59:48): And so it's non obvious and it depends on the terrain and it's all sorts of other stuff. And so eventually I think you want as much coverage information as you can. And just like telcos struggle with this problem and make up maps, it's just a never-ending problem, but I think distributing it and opening it is really the best way to make it happen.
Arman Dezfuli-Arjomandi (01:00:12): Well, it's going to be fascinating to watch how the network plays out. It's certainly been an exciting year. I think we're coming up on just one year. I believe the first block was mined July 29th, right? 2019. Does that sound right?
Amir Haleem (01:00:24): Yeah, that's exactly correct. So I think we've got another 12 days until the anniversary and yeah, it's been interesting, certainly. We've learned an absolute ton. But I'm really happy and certainly very proud of the team in terms of what's been built here. I think it's really one of the most complete implementations of a valuable or potentially useful blockchain network. And I hope that we see a lot more.
Arman Dezfuli-Arjomandi (01:00:51): It's insane how fast you guys have moved and how many iterations you've done and how the whole thing hasn't come crashing down in the meantime. I mean, it's just, it's been really a thrill to watch and I just can't wait to see what comes next.
Amir Haleem (01:01:05): Yeah. Thank you. I think, like I said, I think we've gone, the team has gone in an extraordinary pace, and it's exciting. There's a lot to do. And like I said, our desire is that we are just one of many contributors to the network, but I think the nature of these things is that someone has to push it into life. And I think we've done that and now we're starting to see like a pretty good ecosystem of stuff forming. So very, very happy with that. I think it's awesome, and we're just excited to watch where the network goes and happy to be a participant in it.
Arman Dezfuli-Arjomandi (01:01:41): Amazing. Well, Amir, thanks for your time. Hope you have a good rest of your day and talk to you again soon.
Amir Haleem (01:01:47): Yeah. Thank you. Talk to you soon.
Arman Dezfuli-Arjomandi (01:01:48): Take care.