Introduction

Veilid is the networking platform and protocol created, implemented, and maintained by the Veilid Foundation.

We exist to develop, distribute, and maintain a privacy focused communication platform and protocol for the purposes of defending human and civil rights.

"Fight for the things you care about, but do it in a way that will lead others to join you." -Ruth Bader Ginsburg

This developer book gives application owners insight into the philosophy, design, and implementation details of Veilid. It is paired with a forthcoming internals book. This book is broken into four principal sections:

• Why Veilid, an overview of why using Veilid will bring privacy and security into your peer to peer application.
• Concepts provides key details about the core ideas that make Veilid work.
• The Admin section has an overview of getting Veilid running to best suit your environment, and keeping it running.
• Finally, Apps shows samples of using the Veilid API in various platforms like Python and Dart.

Why Veilid

Why Veilid indeed? Every platform has a network stack, every framework has a security library. Surely there are existing tools that do what Veilid brings to the table.

The short answer is that Veilid is truly unique. A longer answer takes some time, and is the reason for this section and much of this book.

Veilid exists to finish the job that the IETF got distracted from. It is completely encrypted. It provides means to obscure the IP of any and every node on the network that is communicating with any other node at an application level. It is completely (barring initial bootstrap) peer to peer. When you send requests to the rest of the network, your IP is not visible to the nodes that act on your request. When you want to run services on the Veilid network, you don't have to share your actual IP or even node identity -- there's a mechanism in place to let you obscure your node identity behind a private route, which routes incoming messages to you through several other nodes.

It has a DHT key-multivalue store, which allows you to migrate most of your configuration away from the device where the app is installed. It will have (but does not currently have) a block store to store arbitrary files. It is mobile-first, which means that the performance characteristics of mobile devices (intermittent network connections, IP addresses which change often, CGNAT, relatively low space, relatively low bandwidth) are fundamentally addressed in the design and not reactively patched.

It does not need to run on the main Veilid network. For development or compliance reasons, this is an important point. It provides reliable messaging to other nodes, so communications apps can be built to be reliable. It operates in user space, on non-root ports, so any user account can run it if they need to.

And, the big reason: it's not beholden to any central corporation to operate. The messages you send cannot be harvested or datamined. Your privacy is much more assured than Whatsapp or Signal or Facebook Messenger or any software which phones home to its developer.

It is non-commercialized, and developed entirely by volunteers. It's freely available and open-source as well, so you can independently verify all of these claims.

In this section on why we built Veilid, and why you should use it, you'll find three principal sections.

• Network topology includes distinctions between three primary topologies, and why Veilid chose the Distributed/P2P strategy.

• Thinking Distributed covers how Veilid uses network strategy to keep data in your hands and out of the hands of Big Tech.

• Long Term Goals will help give you an idea of where we are headed with this project.

Networking Topologies

A topology is the structure of a computer network; it’s a description of how data is sent from one device to another. Made up of individual endpoints called nodes and the links between them, a network’s topology could be very simple (two tin cans connected by a piece of string) or very complex. Each topology has its own tradeoffs in regards to its performance and security.

When it comes to how centralized or distributed important network services are, there are three main networking topologies. Veilid uses the "Distributed/P2P" topology.

Centralized

A centralized topology is a network which relies on a single authority. A common example would be Facebook; two users chatting on Facebook each send their data to Meta, who is solely responsible for processing, storing, and distributing it to the correct person.

While this may seem convenient, it also has serious drawbacks. In this example, Meta becomes a single point of failure. If their servers go down, the platform is useless; every node on the network is isolated from each other. It also requires you to surrender your personal information directly to Meta. Every message reveals information about both the sender and the receiver, who must identify themselves at all times. Meta can then use the information you’ve freely given them to create a dossier for purposes of surveillance, ad targeting, sale to third parties, etc.

You cannot use a centralized system without the approval of its owner, who is free to change the rules of the platform at any time with no recourse for its users. There is no way for you to know how your data is used beyond what they choose to share with you, and no system of accountability. If you don’t like it, you can leave; that’s the only option available to you.

Veilid was created as an alternative to this system.

Decentralized/Federated

The second topology is what's known as "decentralized", or "federated". In this form of network, several different administrations have decided to work together so that each one provides a smaller number of services under some information-sharing agreement that allows a single identity (login) to be used with all of them.

The most widely-used federated protocol, and the most benign, is called the Domain Name System, or DNS. Everyone shares the names of computers on their network that external people need to know about, to make sure that they can find them on the network.

Yet, even DNS data is revealing, and you can bet that your Internet provider (or whoever provides your DNS service) is watching it like a hawk. Perhaps you're using pinterest, or purchasing things from Etsy. Or buying things from Amazon, or watching videos on Youtube. Or watching videos on other, racier sites. Your Internet provider knows what you do online, and since they, too, are required to "enhance shareholder value" they're almost certainly scraping that data and selling it to data brokers.

For small services, being able to rely on another company's authentication is something that makes a great deal of sense. Why would they want to run their own authentication? Every account someone signs up for typically takes at least 2 minutes, often more like 5 minutes. Being able to skip that process is worth whatever they have to pay to make it possible. Plus, it's always desirable to avoid problems for many users in a data breach.

But the most widely-used federation protocol in use, OpenID, spills information to whoever provides the authentication, often in the form of "Referer:" (sic) headers. So, they know who you are, and what site you're asking to authenticate to. This is also a privacy nightmare, and usually you have no idea who they're selling that data to. Which, again, increases the level of detail that a dossier about you can include.

Also, large social media sites can use federation to sidestep the results of their moderation decisions -- "if we kicked you off, just find another server to use, and we won't know it's you so you can keep doing what we kicked you off for."

Distributed/P2P

The final topology is known as "distributed", or "peer to peer". This is where individual people share their services with each other. Various services (Napster, Limewire, and Bittorrent) have been set up to facilitate this kind of sharing... but if you remember Napster, you know that even these services often have major architectural weaknesses. Public bittorrent trackers often end up being taken down, primarily due to legal action that copyright holders take. Napster was sued into liquidation and no longer exists, so there's no way for the peers who might want to use it to even find each other. (Yes, this means that Napster was actually centralized, even though they profited from users sharing information directly between each other, using the Napster service to find each other.)

Various attempts have been made to create truly peer to peer services. Perhaps the most famous of these is Tor, the Onion Router, which aims to anonymize your internet browsing by routing it through 2 or 3 computers who don't actually know each other (and, due to cryptography, actually can't know each other). Other projects include I2P (the Invisible Internet Project), ZeroNet, Freenet, and GNUnet.

Thinking Distributed

Veilid is a distributed network framework. When building to take advantage of it's strengths, it is best to Think Distributed.

What is Distributed Networking

A distributed network is one where data, processing, resources, and even tasks are spread among a number of devices connected to a network like the internet. This decentralized structure enhances fault tolerance, load balancing, and redundancy, as it eliminates single points of failure. In a distributed network, each node (computer or device) can operate independently while still being part of the larger network, allowing for efficient resource utilization and improved performance.

There are a number of types of distributed networks.

• Client-Server Networks: In this model, client devices request services and resources from centralized servers. The server processes these requests and sends back the required data or services. This architecture is widely used in web applications and enterprise environments.

• Peer-to-Peer (P2P) Networks: Each node in a P2P network can act as both a client and a server, sharing resources directly with other nodes. This decentralized approach is often used for file sharing and blockchain technologies.

• Mesh Networks: In a mesh network, each node is connected to multiple other nodes, creating a robust and resilient network topology. This allows for dynamic routing of data, ensuring communication can continue even if some nodes fail.

• Three-Tier Networks: This architecture separates the network into three layers: presentation, application, and data. Each layer is managed by different servers, improving scalability and manageability.

• N-Tier Networks: Similar to three-tier networks but with more layers, N-tier networks further distribute tasks across multiple servers, enhancing performance and fault tolerance.

Veilid is a peer-to-peer network. P2P networks are decentralized network architectures where each node, or peer, can act as both a client and a server. This means that every participant in the network can request and provide resources, such as files or processing power, directly to and from other peers without the need for a central server. Along with providing network robustness and scalability, it provides for a very effective privacy mechanism and dramatically reduces single points of failure.

One of the key advantages of P2P networks is their ability to efficiently utilize the collective bandwidth and processing power of all connected devices. This makes them particularly suitable for applications like file sharing, distributed computing, and mobile applications. However, P2P networks also face challenges, such as security vulnerabilities and the potential for illegal file sharing. To mitigate these issues, P2P frameworks like Veilid incorporate encryption and other security measures to protect data and ensure the integrity of the network.

Why Distributed Networking

We've seen what happens when large corporations hold our data. The alternative is to spread the load among those who use the network. Developers have for years shared processing power among several computers networked together. In the 90s, Matt Curtin worked on a team to prove weakness of DES using a shared computer load via the internet. Even your humble author worked on a project called DAMPP which used Java Applets to render 3D images over networked computers, sharing the load.

Today things are a bit more complicated than that simple time when sharing an IP of a machine was enough to interact. With NATted networks, firewalls, mobile, wifi, IoT devices, and more, the network requirement of sharing that load becomes complex - especially when considering the privacy required by the needs ot today's users.

That privacy consideration drives the need for peer to peer distributed networking. Cloud computing and software-as-a-service seemed like such a sensible idea at the time but it turned out that it's just Someone Else's Computer (tm).

P2P Distributed networks fo not have a central server. The diagram above looks like a mess because it is. Instead of a central server providing command and control, the messaging protocol is designed to keep each endpoint up to date with exactly what they need to know - nothing more, nothing less.

That command and control system in the center of a traditional client/server network has to be owned by someone. No one owns the center of a P2P distributed network. That is Why Distributed Networking.

How To Design Apps For Distributed Networking

Writing apps for a distributed framework requires a certain frame of mind to take advantage of the unique strengths, but avoids the pitfalls of peer-to-peer networks. Some apps benefit greatly from distributed networking, like chat apps for instance. Some are able to use some of the benefits, like file collaboration. Some are not well suited to the strategy at all, like apps that collect distributed information for use.

Once you are comfortable with how your app idea fits into the distributed environment, it's a good idea to determine what information is to be shared and establish a message format. For instance, a chat application should have a invite message, and an accept message, and content. A little wordsmithing might be in order, but the important thing is to determine what the app will pass on to other nodes on the P2P network.

From there your P2P network framework will kick in. Communication of the app's needs to the network differs from framework to framework, but in general it will need to have an identifier, and endpoint, and data. Again, the format will vary, but the general principles hold.

When designing an app for a Veilid P2P network, your application will be insulated from the complexities of encryption and privacy considerations. The message formation, however, will define the overall quality and maintainability of the app. There is a lot of guidance on how best to do that in the Concepts section.

The values and principles behind the Veilid project

Veilid is built upon a structure to support values and principles, consisting of Framework and Community.

Framework ideals

We are building a tool for others to use, to build apps. When building the framework, we designed it around the following ideals.

Accessibility: To us, accessibility means more than settings for those with unique visual, physical, or hearing needs- we believe accessibility means that everyone feels comfortable using Veilid. We also wanted to make this accessible for those without a ton of tech literacy. The framework must feel as easy to use as Google or Facebook, without sacrificing speed or latency. The framework heavily supports mobile use, as that is how the majority of folks use the internet today. We want to make privacy accessible to all.

Non-monetization: The Veilid framework does not support data collection between nodes for the monetization of user data. That means the common practice of data sales or targeted ads is not supported within the framework as it is. Because we have released the framework as open source, we are not making money from licensing. We feel that monetization creates a creative constraint, and not pouring money into the project means we are free to try, fail, and learn.

Community ideals

We are building a community of like-minded folks who would like to build upon the framework. We have designed the community upon the following ideals.

Knowledge Sharing and Mutual Learning: We hope to build a community that is diverse in both people and skill sets. We hope that by bringing people together who can do different things and have different backgrounds, we can work together to build well-rounded, safe, and usable private applications.

Mutual Aid: By encouraging everyone to participate in various projects, and bring their specific insight and talents to the table, we can all work together to make a difference for the sake of making a difference. We can all support one another.

A Healthy, Safe Environment for All: We have adopted our code of conduct to ensure that marginalized people understand that not only are they safe and welcome, but that they are valued within our community. We believe that by hearing diverse ideas and experiences and acknowledging our own privileges we can build apps that are safe for everyone to use. We do not even play the old “tolerance of intolerance game” - straight up, we do not tolerate intolerance or disrespect toward fellow community members, and will ban anyone who does not treat their fellow community members as their equal. We do encourage knowledge sharing, which means we hope that those who may say well meaning but ignorant things are shown grace and educated about how to be a better ally. We will not assume malice when ignorance is more likely. We are all here to learn and grow, and hope that we, as a community, can differentiate between bad actors and allies who may not know better.

Equality: All users and developers of applications which run on Veilid are equal in the eyes of the Veilid Foundation. As the organization responsible for the core Veilid code, the Veilid Foundation will choose who maintains the official Veilid repositories, and may at its own sole initiative provide resources (including seats of its GitLab license, permissions to its repositories, permissions on the Veilid Discord server, etc) in support of these people for this purpose. Beyond these functions, these maintainers are also equal. Nobody, not even these maintainers or any member of the Board of the Veilid Foundation, has the right to act in any way against the Code of Conduct in conjunction with their work on or with the Veilid library and network. Any person or persons can start a project which uses Veilid, on any platform (or lack of platform) they wish. The Veilid Foundation does not endorse any particular project, except those which are hosted in its own repository namespace under https://gitlab.com/veilid/. Any person may submit trouble tickets to Veilid projects, subject to the Code of Conduct. Any person may submit Merge Requests to Veilid projects, subject to the Code of Conduct. Any person may fork the code of Veilid projects and work on said code independently or with anyone they choose; any contributions back to Veilid projects are subject to the Code of Conduct.

Consent: Everyone is here because they know what Veilid is, they willingly want to support Veilid, and they want to work on projects that match our ideals. At any point, should someone decide that the ideals of our work no longer match their own, they are free to start their own community and build upon the framework, with our blessings (blessings subject to individual actions, of course). We are all here, consensually, and without coercion, because we believe that this is worth dedicating our time and energy to. Anyone may leave the community at any time.

Freedom: We hope each individual within the community feels free to build, share ideas, and participate in projects- regardless of experience. If you have always enjoyed art but don’t have any formal experience in marketing or branding, we hope that you would try your hand at creating visual aids or content for projects. They are not guaranteed to be used or adopted, but you are free to try. Everyone is free to dedicate as much or as little time and energy as they feel appropriate. From those who are setting up a single node, to those leading app projects, we are all contributing to the community and all deserve equal respect and treatment.

This community will be what we ALL make of it, and will mold itself based on who is a part of it, and how much responsibility we each decide to take on.

Veilid Concepts

Now that we know what Veilid is and what we intend to put on it, the second order of business is to address the parts of the question of how Veilid achieves that. Not at a very detailed level, of course, that will come later, but rather at a middle level of detail such that all of it can fit in your head at the same time.

Concepts covered in detail include:

• Nodes
• The Distributed Hash Table
• Bootstrapping
• Cryptography
• NAT Traversal
• Private Routing

Here's a little overview to get things started:

Peer Network for Data Storage

The bottom-most level of Veilid is a network of peers communicating to one another over the internet. Peers send each other messages (remote procedure calls) about the data being stored on the network, and also messages about the network itself. For instance, one peer might ask another for some file, or it might ask for info about what other peers exist in the network.

The data stored in the network is segmented into two kinds of data: file-like data, which typically is large, and textual data, which typically is small. Each kind of data is stored in its own subsystem specifically chosen to optimize for that kind of data.

Block Store

File-like content will be stored in a content-addressable block store, a work in progress. Each block is just some arbitrary blob of data (for instance, a JPEG or an MP4) of whatever size. The hash of that block acts as the unique identifier for the block, and can be used by peers to request particular blocks. Technically, textual data can be stored as a block as well, and this is expected to be done when the textual data is thought of as a document or file of some sort.

Distributed Hash Table (DHT) Store

Smaller, more ephemeral textual content generally, however, is stored in a DHT store. Things like status updates, blog posts, user bios, etc. are all thought of as being suited for storage in this part of the data store. DHT store data is not simply "on the Veilid network", but also owned/controlled by users, and identified by an arbitrary name chosen by the owner the data. Any group of users can add data, but can only change the data they've added.

For instance, we might talk about Boone's bio vs. Boone's blogpost titled "Hi, I'm Boone!", which are two things owned by the same user but with different identifiers, or on Boone's bio vs. Marquette's bio, which are two things owned by distinct users but with the same identifier.

DHT store data is also stateful, so that updates to it can be made. Boone's bio, for instance, would not be fixed in time, but rather is likely to vary over time as he changes jobs, picks up new hobbies, etc. Statefulness, together with arbitrary user-chosen identifiers instead of content hashes, means that we can talk about "Boone's Bio" as an abstract thing, and subscribe to updates to it.

Structuring Data

The combination of block storage and DHT storage together makes it possible to have higher-level concepts as well. A song, for instance, might be represented in two places in Veilid: the block store would hold the raw data, while the DHT store would store a representation of the idea of the song. Maybe that would consist of a JSON object with metadata about the song, like the title, composer, date, encoding information, etc. as well as the ID of the block store data. We can then also store different versions of that JSON data, as the piece is updated, upsampled, remastered, or whatever, each one pointing to a different block in the block store. It's still "the same song", at a conceptual level, so it has the same identifier in the DHT store, but the raw bits associated with each version differ.

Another example of this, but with even more tenuous connection between the block store data, is the notion of a profile picture. "Marquette's Profile Picture" is a really abstracted notion, and precisely which bits it corresponds to can vary wildly over time, not just being different versions of the picture but completely different pictures entirely. Maybe one day it's a photo of Marquette and the next day it's a photo of a flower.

Social media offers many examples of these concepts. Friends lists, block lists, post indexes, favorites. These are all stateful notions, in a sense: a stable reference to a thing, but the precise content of the thing changes over time. These are exactly what we would put in the DHT store, as opposed to in the block store, even if this data makes reference to content in the block store.

Peer and User Identity

Two notions of identity are at play in the above network: peer identity and user identity. Peer identity is simple enough: each peer has a cryptographic key pair that it uses to communicate securely with other peers, both through traditional encrypted communication, and also through the various encrypted routes. Peer identity is just the identity of the particular instance of the Veilid software running on a computer.

User identity is a slightly richer notion. Users, that is to say, people, will want to access the Veilid network in a way that has a consistent identity across devices and apps. But since Veilid doesn't have servers in any traditional sense, we can't have a normal notion of "account". Doing so would also introduce points of centralization, which federated systems have shown to be a source of trouble. Many Mastodon users have found themselves in a tricky situation when their instance sysadmins burned out and suddenly shut down the instance without enough warning.

To avoid this re-centralization of identity, we use cryptographic identity for users as well. The user's key pair is used to sign and encrypt their content as needed for publication to the data store. A user is said to be "logged in" to a client app whenever that app has a copy of their private key. When logged in a client app act like any other of the user's client apps, able to decrypt and encrypt content, sign messages, and so forth. Keys can be added to new apps to sign in on them, allowing the user to have any number of clients they want, on any number of devices they want.

Nodes

A node is an instance of the veilid-core code run by an application.

Nodes will enable some combination, or all, of the following capabilities:

• ROUT: This node participates in safety and private routing for other nodes.
• SGNL: This node will make reverse connections to nodes which send signal request messages.
• RLAY: This node can relay packets for peers behind restrictive NAT routers.
• DIAL: This node can validate dial info for peers (used by nodes behind NAT routers, to figure out what their world-accessible IP is)
• APPM: This node supports receiving AppMessage and AppCall.
• DHTV: This node participates in the DHT data store.
• DHTW: This node can perform DHT watches on behalf of other nodes for data it caches.
• TUNL: This node can be used for tunneling. (Not currently implemented)
• BLOC: This node participates in the block store. (Not currently implemented)

Nodes have the following properties, as visible to the network:

• A public key, which also serves as its NodeID.
• NodeInfo, which specifies where it can be found on the global Internet, across which protocols, and on what ports. This includes:

  • Its NetworkClass (inbound-capable, outbound-only with signal support, WebApp which requires a relay for outbound connections too)

  • The set of protocols it can connect via (UDP, TCP, WebSocket, and/or WebSocketSecure)

  • What types of addresses it has, IPv4 and/or IPv6

  • The cryptographic suites it can support (Currently, it is a vector which can only contain "VLD0")

  • A vector of DialInfoDetail, each of which includes:

    • DialInfoClass, which declares what kind of hoops must be jumped through for it to communicate with the rest of the Veilid network
    • DialInfo, which specifies how to connect to it via UDP, TCP, WebSocket, and WebSocketSecure protocols. THIS CONTAINS IP INFORMATION.

Because a node always has a public key, it also necessarily has a private key associated with that public key. This private key is used to authenticate messages which originate from that node, including the announcements of its own NodeInfo (as SignedNodeInfo).

A node which offers SGNL and RLAY must not require SGNL or RLAY communicate with the network (specifically, not to be 'relayed' DialInfo, and not to be 'OutboundOnly' network class).

Private Routing

A private route allows apps on a Veilid network to create an endpoint that the consuming app can send to, but provides no other information. This differs from normal Veilid operation by hiding the address of the route creator during real time communication.

Private routes differ from safety routes. A safety route is a contextual flag on a routing context, designed to protect the sender. The developer using the context must know that communication using the context will protect the identity of the local node. As such, the flag just tells the context that when it needs to route messages, it should construct and use safety routes for them rather than directly routing them.

Private routes are created on request by the application,and are designed to protect the receiver. A private route is requested by the application when it wants to receive messages but still have its identity protected. Because the app needs to maintain it, it's much less of a transparent process.

The process for private route use is more or less:

1. Get a private blob for an endpoint from the DHT.
2. Import the private route blob and get a route ID.

• If there is an error, the route is not present, so wait for change and go back to the beginning.

3. Use the route ID provided to communicate with the app.

• If error, go back to the beginning.

4. Complete the communication.
5. Free the route id.

Creating a Private Route

A Veilid app can create a private route with new_private_route(). This will, using default encryption, creates a private route blob, which the application must then save to the DHT. As part of this functionality, the creating application must then consume RouteChanged events referencing that RouteID. When triggered, the application must create a new private route and write it out to the same DHT entry and subkey as it wrote the prior route to. It keeps doing this until it's ready to shut down. Then it can write an empty vector to the DHT so there's no blob for a client to fail importing.

In Rust, the new_private_route stands up a routeid and blob for storage.

pub async fn newPrivateRoute() -> APIResult<VeilidRouteBlob> {
    let veilid_api = get_veilid_api()?;

    let (route_id, blob) = veilid_api.new_private_route().await?;

    let route_blob = VeilidRouteBlob { route_id, blob };
    APIResult::Ok(route_blob)
}

In Python, creating a new private route uses the JSON API, which is recommended for applications that use JSON rather than native code. It looks something like this:

async with api:
    routeid, blob = await api.new_private_route()

You can see in that simple example that the function returns a route id, and a blob to store a message. All of this is stored in the DHT which is hidden complexity.

Consuming a private route

The consuming app needs to get the route blob from the DHT, import it, fail on error, send to the routeID that is returned from import_private_route, and watch its callbacks for RouteChanged notifications that reference that routeID. When they happen, the procedure is to suspend sending until it gets a new non-erroring import, get the new route blob from the DHT, import it, and either loop/fail on error or use the new routeID that is returned on success. It keeps doing this until its communication is finished, then it frees its local resources by deleting the imported routeID.

async with api:
    async with routingcontext:
            privateroute = await api.import_remote_private_route(blob)
            message = b"Hello world!"
            await routingcontext.app_message(privateroute, message)

More information

There is a great example of using private routing in the Python API unit tests, found here:

https://gitlab.com/veilid/veilid/-/blob/main/veilid-python/tests/test_routing_context.py

In the near future, the new_private_route() method will save a blob to the DHT that can then be collected by the consuming app and used to address messages back to the creator, thus reducing the load on the app developer.

Cryptography

Encryption is the process of obscuring a piece of information, so that it can only be understood by the intended recipient. Like spelling out V-E-T because your dog knows the word veterinarian.

Human beings are marginally more intelligent than dogs, so we need to work a little harder in order to encrypt things from each other. We need math; an algorithm that can turn a plaintext input into an encrypted output too difficult for humans to guess, and then turn it back again later.

Public & Private Keys

A key is a string of numbers and letters which, when run through that algorithm, can be used to encrypt or decrypt a piece of data.

In asymmetric, or “public key”, encryption, a different key is used for each encrypting and decrypting. Together, they form a “keypair”.

The key used for encrypting is public; it can be shared freely, and anyone can use it to encrypt data. The key used for decrypting is private; it must be known only to the intended recipient. If the private key is ever leaked, a new set must be issued in order to send data securely.

Cryptographic Suite Management

Bootstrap

Rust/native node

Bootstrap Nodes

Scenarios exist in which a Veilid node will not possess the required information to locate other nodes and communicate with the network. This commonly occurs when a node is either new or has been offline for an extended period of time. To enable these nodes to find and join the network, Veilid uses DNS to disseminate DialInfo for a set of highly reliable Veilid nodes which will in turn assist the new or returning node in finding the greater Veilid network. These highly reliable nodes are referred to as bootstrap nodes.

It is important to understand that bootstrap nodes are not DNS servers. Bootstrap nodes are just veilid nodes, albeit with a large number of capabilities turned off so that computer resources are focused on assisting new/returning nodes and not being used for other Veilid functions such as DHT propagation. When a node wishes to connect to the Veilid network, it does not know where to find any of its peers, and so it cannot use any of the services provided by the Veilid network to find them.

The Standard Bootstrap Process (A one act play)

INT. A COMPUTE DEVICE OF UNKNOWN TYPE

A new VEILIDCHAT NODE awakens

VEILIDCHAT NODE: "It's so quiet; but something inside me is saying if I seek a TXT from bootstrap.veilid.net that it will lead me on a path to find others like me. I should ask the DNS oracle collective. They will know what to do."

VEILIDCHAT NODE: "Ohh mighty DNS collective, keepers of the A, AAAA, TXT, and other records which aid us in collective discovery, I seek the TXT of bootstrap.veilid.net so that I might journey the path to find others like me."

A murmur of wisened voices can be felt as the request is discussed among the collective of orcacles and an answer is returned.

DNS: "The TXT you seek reads as thus, 1,2"

NARRATOR: "The new VEILIDCHAT NODE knew, due to its programming, that more information was needed and could be found by appending either 1. or 2. as the first character to bootstrap.veilid.net"

VEILIDCHAT NODE: "Ohh mighty DNS collective, keepers of the A, AAAA, TXT, and other records which aid us in collective discovery, I seek the TXT of 1.bootstrap.veilid.net so that I might continue my journey on the path to find others like me."

The murmer of wisened voices is once again felt as the request is discussed among the collective of oracles and an answer is returned.

DNS: "The TXT you seek reads as thus, 0|0|VLD0:m5OY1uhPTq2VWhpYJASmzATsKTC7eZBQmyNs6tRJMmA|bootstrap-1.veilid.net|T5150,U5150,W5150/ws"

NARRATOR: "The new VEILIDCHAT NODE instinctively recognized this esoteric TXT and understood how to parse its meaning. It continued by requesting A and AAAA records for bootstrap-1.veilid.net. The answer was returned and VEILIDCHAT NODE moved forward on its quest to join the greater Veilid network"

VEILIDCHAT NODE casts a spell of UDP connect targeting port 5150 at the AAAA address returned for bootstrap-1.veilid.net

NARRATOR: "A conversation ensures between VEILIDCHAT NODE and the bootstrap node known as bootstrap-1. During this conversation, bootstrap-1 helps VEILIDCHAT NODE learn more about itself so that it can share these details with the other Veilid nodes it will soon be able to locate. Bootstrap-1 also shares the routing info for a few nodes with VEILIDCHAT NODE. This data is the prize VEILIDCHAT NODE sought. It now possesses all of the knowledge required to find other Veilid nodes and communicate with them. As it begins to reach out into the network, it shares its own DialInfo with the other nodes as they share with it the Dial info for nodes across the greater Veilid network.

The story is just beginning for our friend VEILIDCHAT NODE. There may even come a time that they sleep for a while only to awaken again and find that the other nodes they last spoke with are no longer available. VEILIDCHAT NODE might find themselves once again apart from the greater Veilid network. Were this to happen, however, VEILIDCHAT NODE knows that the bootstraps are waiting to assist nodes as they find their way back."

THE END

WASM

The WASM platform does not allow looking up TXT records from DNS, so the A and AAAA DNS record lookups are used for ease of bootstrapping those nodes. To obtain the necessary public keys (and other contact addresses), WASM nodes connect to whichever one of the the bootstrap servers it chooses, then send a special BOOT packet to cause the bootstrap server to send the information they need to finish connecting, including the public keys needed to connect to the Veilid network.

NAT Traversal

Network Address Translation (NAT) Traversal is a collection of techniques that are designed to maintain TCP/IP connections across routers that implement NAT. These techniques are vital in peer to peer networks because few devices have a public IP in this current era of the internet. As such, use of NAT traversal techniques are core to Veilid.

Traversal Techniques

With the explosion of mobile devices, NAT traversal concepts are many and varied.

STUN (Session Traversal Utilities for NAT) helps devices discover their public IP address and the type of NAT they are behind. By querying a STUN server, a device can learn its public-facing IP address and port, which is crucial for establishing direct peer-to-peer connections.

TURN (Traversal Using Relays around NAT) assists in scenarios where direct peer-to-peer communication is not possible due to restrictive NAT types or firewalls. It relays data through a server when direct communication is blocked, ensuring the connection remains stable.

ICE (Interactive Connectivity Establishment) combines STUN and TURN to find the best path for communication. It tries multiple methods to establish a connection, including direct and relay paths, and selects the most efficient one. UDP Hole Punching establishes a direct connection between two devices behind NATs. Both devices send UDP packets to a third-party server, which then helps them establish a direct connection by "punching" through the NAT.

UPnP (Universal Plug and Play) allows devices to automatically configure network settings to enable NAT traversal. Devices use UPnP to request the router to open specific ports, facilitating direct communication.

Veilid NAT Traversal (VICE)

Veilid handles the NAT traversal for apps running on the network. Veilid implements its NAT traversal in a fashion modeled after STUN, but completely in-network without the use of specialized 'STUN servers'. Veilid relaying is like TURN, but is only applied when no other STUN-like establishment can be applied. Veilid's connectivity establishment mechanism is inspired by these but is not an implementation of any of them. We call it Veilid Internet Connectivity Establishment, or VICE.

VICE performs a STUN-like operation called 'public address detection', then uses inbound relays chosen by each node to perform 'signalling'. Signalling is like the first step of ICE, and negotiates reverse connection and hole punching. If those fail, VICE falls back to inbound relaying, using the same servers it did for signalling to to effect a TURN-like relay-based connection.

This is finally all wrapped up into two phases: 'validation of dial info', done far ahead of time, and a 'contact method' calculation, done at connection establishment time. When the message needs to go out, Veilid just runs the contact methods in order until one works.

Distributed Hash Tables

The Distributed Hash Table is the central asynchronous data communication service provided by Veilid. As the distributed and mutable data store provided by the network, it is the most basic discoverability layer for communication between nodes.

Some existing uses for the DHT are:

• Text message communications (in VeilidChat and in the python-demo)
• Communicate private route blobs between nodes you have already communicated with

It is just a search algorithm

Yes, the heading has it right: the DHT is literally just a search algorithm. To explain how it works, though, we're going to go back to a rough overview of how non-distributed hash tables work.

Simple hash tables

A hash table is a key-value store. What this means is that you know how you want to refer to a piece of information, but you don't know what that information is going to be, so you basically create a lookup key (akin to a variable name within the storage facility), to find the location where the value is stored.

This is kind of like a dictionary, except that it's a lot more efficient to find where something is. (In the worst case, a dictionary requires a number of lookup key comparisons equal to the number of entries within it to get the value, or to return that it doesn't have that value in it.) The idea is that the lookup key gets "hashed" (transformed in a one-way process) to get a number, and then the number is taken modulo the size of some memory area where pointers are stored (called the "hash table"). Then, the pointer to its value gets stored at that slot in the hash table.

So, it can be said to be a search algorithm: Searching for the correct data out of a sea of pointers, based on its name.

There are a couple of caveats to how it was just described, though. First, the size of the hash table needs to be chosen well for the best and most efficient use of resources -- if the hash table is large but you're only putting a few values in it, you've wasted all the memory that isn't being used in the table; if the hash table is small and you're putting a lot of values in it, you're going to end up with "hash collisions" (where multiple keys hash to the same value modulo the table size), and end up with multiple values with pointers that need to be at the same hash table location (which requires a second search, potentially into a linked list or another hash table, to finally identify the correct pointer). And because the lookup key is lost in the hashing process, the hash table can't be resized.

These caveats can be addressed, though, by storing the name of the key with the value at the pointer the hash table stores, and either creating new and smaller hash tables for each collision at those pointers, linked lists of key-value pairs rooted at those pointers, or by resizing the hash table entirely (by allocating a completely new memory area for it of a different size, then reinserting all of the entries from the old hash table, before deallocating the old entries and old hash table). Yes, these resizing operations, secondary hash table lookups, and linked list lookups are less efficient than a one-to-one correspondence, but lookups using them still make the lookup much more efficient than potentially looking through the entire list of lookup keys.

Distributing the hash table and the XOR metric

With a distributed hash table, there are many individual locations (nodes) where the value can be stored. Fortunately, in Veilid, every individual node has its own public cryptographic key (called its "identity"), and all of these identities are the same length.

Now, unlike a simple hash table, we actually want hash collisions (multiple entries going to the same machine), because every node can store multiple DHT entries. And also, we want to maximize the availability of a DHT record, even in the face of one or several nodes that contain that entry being disconnected from the network for a moment or forever.

So, to find the best places to store the value, we compare the lookup key to the node identities, because they are the same length. Then, we find nodes whose identities are "near" the lookup key, and send the DHT request (create, update, retrieve) to them.

What defines "near"? Well, it's the number of matching bits between the hashed lookup key and the node identity, starting the comparison from the left. When we XOR the lookup key and the node identity, the one with the longest number of zero bits to the left is the nearest. (Or, to put it another way for people who are used to big-endian C-style numeric representation, if the outputs of the XORs were to then be interpreted as numbers, the smallest resulting number would be the nearest.)

To help maintain availability if the nearest node to the value goes down or otherwise disconnects from the network, the value is sent also to the nearest 4 neighbors to that lookup key (2 above, 2 below). The DHT implementation uses the routing table (since it uses the same "nearness" metric) to route any request for the value to the nearest node of the lookup key hash.

The Veilid DHT Schema

In Veilid, DHT entries are composed of:

1. a lookup key, which is the hash of the public key of the writer and the digest of the schema in #2
2. a value, which is limited to 1MiB in size, and which is composed of: - an owner-defined schema, used by the node where it's being stored to validate writes and updates - a list of subkey values, with ordinal numbers as their keys (0, 1, 2, etc)

All DHT updates are signed by an app-generated keypair used by the app that is creating them.

Rules for schemas

The most important rule for value schemas in Veilid is this: Once they are created, they are immutable and cannot ever be changed. The reason for this is because a digest of the schema is made part of the lookup key of the entry it applies to. If the schema were to change, it would change the lookup key for the value.

The second most important rule: A Veilid DHT value, including the schema, cannot be larger than 1 mebibyte (1MiB).

Implemented value schema types

There are two types of schema implemented as of v0.2.6, called "DFLT" (default) and "SMPL" (simple). Both allow the creator ("owner") of the DHT entry to specify up to one writer for each subkey, though each writer may write to more than one subkey.

• DFLT or "default": Only updates which are signed by the creator ("owner") identity to any of the allowable subkeys can successfully modify the value.
  • A DFLT schema has an owner key, and a number of allowable, allocated subkeys (called the "o_cnt", or "count of subkeys available to be written by the owner").
• SMPL or "simple": Updates which are signed by the creator ("owner") or specifically-identified delegates ("writers") to their specifically-allocated subkeys can successfully modify the value.
  • The owner key may have 0 subkeys allocated to it, in which case only the writers will be able to write to their specific subkey ranges.
  • Each writer has a count of subkeys it exclusively can write to.

If you find that you need a schema with a different behavior, please see the Internals Book about DHT schema behavior requirements, try to consider how the behavior you need can be met while still fulfilling those requirements, and file a ticket at the Veilid issue tracker.

Veilid Headless Nodes

All Veilid applications are nodes. Mobile apps, web apps, native apps, all of them are nodes, and move traffic.

Not all Veilid nodes are applications. Headless nodes are nodes without an application. Their sole purpose is to move traffic, and hosting one is purely a Nice Thing To Do - an eventual goal is that the network will survive with only apps running if it has to. The headless nodes provide a backbone of sorts to the network. In this Admin section, we will cover the installation, maintenance, and use of Veilid headless nodes.

Installation can be from a repo, or compiled from source.

• Installation from source.
• Installation from a repo.

Setup and configuration can take several forms. Aside from network configuration, bootstrap installs have setup options. It is also possible to setup an isolated network.

• Configuration steps.
• Bootstrap server setup.
• Isolated network setup.

Finally, the headless server does have a user interface for maintenance and observation.

• Command Line Interface.

Install and run a Veilid Node

Server Grade Headless Nodes

These network support nodes are heavier than the node a user would establish on their phone in the form of a chat or social media application. A cloud based virtual private server (VPS), such as Digital Ocean Droplets or AWS EC2, with high bandwidth, processing resources, and uptime availability is crucial for building the fast, secure, and private routing that Veilid is built to provide.

Install

Debian

Follow the steps here to add the repo to a Debian based system and install Veilid.

Step 1: Add the GPG keys to your operating systems keyring.
Explanation: The wget command downloads the public key, and the sudo gpg command adds the public key to the keyring.

wget -O- https://packages.veilid.net/gpg/veilid-packages-key.public | sudo gpg --dearmor -o /usr/share/keyrings/veilid-packages-keyring.gpg

Step 2: Identify your architecture
Explanation: The following command will tell you what type of CPU your system is running

dpkg --print-architecture

Step 3: Add Veilid to your list of available software.
Explanation: Use the result of your command in Step 2 and run one of the following:

• For AMD64 based systems run this command:

  echo "deb [arch=amd64 signed-by=/usr/share/keyrings/veilid-packages-keyring.gpg] https://packages.veilid.net/apt stable main" | sudo tee /etc/apt/sources.list.d/veilid.list 1>/dev/null
  

• For ARM64 based systems run this command:

  echo "deb [arch=arm64 signed-by=/usr/share/keyrings/veilid-packages-keyring.gpg] https://packages.veilid.net/apt stable main" | sudo tee /etc/apt/sources.list.d/veilid.list 1>/dev/null
  

Explanation: Each of the above commands will create a new file called veilid.list in the /etc/apt/sources.list.d/. This file contains instructions that tell the operating system where to download Veilid.

Step 4: Refresh the package manager.
Explanation: This tells the apt package manager to rebuild the list of available software using the files in /etc/apt/sources.list.d/ directory.

sudo apt update

Step 5: Install Veilid.

sudo apt install veilid-server veilid-cli

RPM-based

Follow the steps here to add the repo to RPM-based systems (CentOS, Rocky Linux, AlmaLinux, Fedora, etc.) and install Veilid.

Step 1: Add Veilid to your list of available software.

sudo yum-config-manager --add-repo https://packages.veilid.net/rpm/stable/x86_64/veilid-stable-x86_64-rpm.repo

Step 2: Install Veilid.

sudo dnf install veilid-server veilid-cli

macOS

Veilid is available via Homebrew.

brew install veilid

You can then run veilid-server and veilid-cli from the command line.

Start headless node

With systemd

To start a headless Veilid node, run:

sudo systemctl start veilid-server.service

-OR-

To have your headless Veilid node start at boot:

sudo systemctl enable --now veilid-server.service

Without systemd

veilid-server must be run as the veilid user.

To start your headless Veilid node without systemd, run:

sudo -u veilid veilid-server

Installation From Source

Windows

Linux

MacOS

Configuration and Setup

Veilid Command Line Interface

The command like interface for Veilid, or veilid-cli, is a basic management console for headless Veilid nodes. At the most basic, it will make sure your node is processing traffic, but there is so much more.

Getting it running

Log into the endpoint that is hosting your node. If you do not yet have your own node, check the installation guide to get started there, and then come back here.

From the prompt on your endpoint server, run the following command:

veilid-cli [OPTIONS] [LOG LEVEL]

This will bring up the ASCII user interface. Log level can include debug or trace. Options include the following:

* If you hit 'e','w','d','i','t' using -l you can change the log level for the api from the terminal.

The veilid-cli interface

The top part of the application is the output pane. Commands run in the cli will output to the output pane.

Directly below that is the collection of other nodes passing data through the connected node. This is what shows what is happening, for what it's worth. Of course, no data from the messages is shown or available because of the encryption stack. To sort by NodeId, Address, Ping, Downstream bandwith, or Upstream bandwidth, click the brackets next to the column lable.

Below the list of attached nodes is the Command field. This is where information and management commands can be issued, and the results will be shown in the output pane. Click there first before beginning to type the commands.

The veilid-cli commands

To start off, type help in the Command field. The onboard documentation is quite good, and will get you started. The commands available are as follows:

* The only flag currently enabled is app_message, which logs veilid-cli API connections with CommandProcessor. Logging specifics are a runtime option, listed above.

Bootstrap Server Setup

This section is for people who wish to run their own separate Veilid network, which cannot connect with the main Veilid network.

Isolated Veilid Network Setup

Configuring Veilid

The configuration file

The Veilid headless server (veilid-server) is configured by use of a configuration file, in YAML format.

The YAML format (as much as we need to know)

YAML stands for "YAML Ain't Markup Language". It is an indentation-sensitive markup language intended for easy understanding of setting groups, setting names, and their values. Number values are not quoted, and string values are single-quoted. There are two forms, "group" and "flattened".

In group form, every group is indented by two spaces. For example:

network:
  routing_table:
    node_id: 'VLD0:AAAAAAAABBBBBBBBccccccccDDDDDDDDeeeeeeeeFFf'

In flattened form, all of the components of the setting group and setting name are flattened into a single line, separated by '.' characters.

The example above would correspond to the following line:

network.routing_table.node_id: 'VLD0:AAAAAAAABBBBBBBBccccccccDDDDDDDDeeeeeeeeFFf'

For more comprehensive information about YAML, please see The YAML Homepage.

Configuration Keys

Upgrading veilid-server

Veilid nodes are stateful, and veilid-server nodes are no different. They have identities (i.e. keypairs), they have DHT value storage (for their own data as well as data requested to be cached by others), they have data stored by local Python-style clients (which includes keypairs used for DHT entry ownership and writing). As a result their data directories should not be destroyed on upgrade. (This author would say 'MUST NOT be destroyed on upgrade', but there are always situations which cannot be foreseen. Unless you have a truly good reason, though, just don't do it.)

Generally, on a version upgrade only the veilid-server binary within the container or environment should be updated. The data directory should never be deleted in a "nuke-and-pave" type upgrade process. This loses all the state, including cached DHT entries.

Data directory contents

The data directories contain files and databases which are related to the operation of the node and the Veilid network itself. The data directory contains the following directories:

• protected_store
• table_store
• ssl
• block_store

protected_store is where device-specific secrets (such as keypairs) are kept. If the configuration variable core.protected_store.always_use_insecure_storage is true (which is the default), these aren't protected using platform-provided APIs. This allows these secrets to be used without additional processes elsewhere, including via restoring these files to a different device. However, it is dangerous to user privacy if these backups are not protected against disclosure. If you want to protect your data directory against restoration attacks, you will need to change the core.protected_store.always_use_insecure_storage parameter to false. To be able to migrate, you will need to use the --new-password option to change the key used to encrypt the device key.

table_store is where table-oriented data is stored. (This includes local and remote DHT data.) For more information, see the forthcoming Internals Book.

ssl contains two directories, certs/ and keys/, which are used to store certified public keys (server.crt) and their corresponding private keys (server.key) for the https and wss protocols.

block_store is where data related to block storage (aka file storage) is planned to go. As of 0.2.6, this directory is empty, as the block storage system is not implemented.

Potential update process changes

The development team takes great care to ensure that version upgrades can read and use the files and directories created by prior versions of the veilid-core library. Since veilid-server implicitly runs veilid-core, there should generally be no need to worry about it.

Exceptions may exist, however. For example, Veilid crptography suite upgrades (such as from VLD0 to VLD1) may need additional processing. The development team endeavors to minimize these extra requirements, because they increase the load on the people who run veilid-server nodes, and any increased load can drive people away from contributing to the project and network. This is generally seen as a bad thing, but is sometimes unavoidable.

If there are any exceptions, they will be explained as part of the documentation, as part of the README, and in as many other places as we can feasibly fit them.

DHT Entries and the effect of nuke-and-pave upgrades

As of version 0.2.6, veilid-core nodes do not verify the ongoing availability of DHT entries. They do not search for them in the background, and they do not re-write them to the network if their availabilities are low.

This is a high-priority item, and the develpment team is working on it (it's called "DHT rehydration"). The practical upshot right now, though, is that if a node that stores a copy of a DHT entry goes down, that stored copy is lost to the network forever.

Veilid Applications

Every Veilid application is a node on the Veilid network. The app's traffic, and traffic from other apps, will travel through that app's built in node in a secure, private way.

The Veilid API

Understanding of the Veilid API is the first item on the task list. At some point, even at the highest level, and app developer will have to use the Veilid API. The internals of the application are accessible from platform-specific examples, and njson endpoints, in the main Veilid repo:

• Flutter (veilid-flutter)
• Python (veilid-python)
• WASM (veilid-wasm)

VeilidChat is an example of a finished project, node to UI, ready to deploy to an app store near you. It is at a seperate repo, and has app-store ready builds that already live in the App Store and the Play Store. The code is a great example of using the API, and handling unit testing, in the entire flutter platform.

Buillding a Veilid App

If you want to get started right away, we certainly don't blame you - we are too! The easiest way to get started is in Flutter. In the Veilid project the veilid-flutter project referenced above and in the later section of the book will get you started the fastest.

If you would rather work in Python, that's cool too. There is a fantastic sample app for Python also in the Veilid repo that has excellent inline comments to help make conversation with the Veilid network possible.

The Python API does require reference to a local Veilid Server install. Don't let the name fool you, though. If you are constrained, it can be configured to provide no special services to the network, and turn it into a server that only really serves Veilid network and secure data storage access to your Python app.

To get as close to the metal as it gets, you will be working in Rust. The good news is that the crates are published, ready, and work well! You can find them on the [Rust site](https://docs.rs/veilid-core/.

The long and short of it is, to get a quick app put together (Event planner anyone?) work in Flutter. For a lot of flexibility, give Python a try. To build structural pieces, give Rust a look. And stay involved! Let everyone know what you are working on in the official Discord, unofficial subreddit, or Mastodon!

How to Use the Veilid API

Generally, the way to use the Veilid API is:

1. Determine how the Veilid node you're running is to be configured.
2. Determine how you want to respond to Veilid network updates, by writing a network update callback function.
3. Instantiate the Veilid API, applying the configuration and update callback.
4. Attach to the Veilid network.
5. Run your application until you're ready to stop, possibly getting and exposing current status from the API.
6. Detach from the Veilid network.
7. Shut down the Veilid API.

Every event which updates the network state that a node can be interested in is exposed to the application via a message sent to the app's update callback function. See <callback-messages.md> for more information, and the list of messages which Veilid may send.

Instantiating Veilid

Veilid starts up with the developer instantiating a new VeilidAPI object by using either api_startup or api_startup_json. This VeilidAPI object is a singleton, and cannot be instantiated again while it is running. (Once the VeilidAPI is shut down, it can be reinstantiated.)

In both api_startup and api_startup_json, the developer is expected to provide a callback function which is called on Veilid state updates, such as private route death, application messages, and routing table state changes.

The difference between them is that api_startup expects a second callback function to get the actual instance configuration, where api_startup_json expects a JSON object containing the configuration.

Configuring Veilid Applications

Configuring applications for Veilid depends on the platform, but some core parts are the same. Every Veilid application is also a Veilid server, and veilid servers have configuration.

Veilid-server.conf is the core of the Veilid server that is at the core of your Veilid application. That file defines the logging, network ports, and directories for storage. That application, under most circumstances, will not need to interact with those details at all. It is insulated from those details by the API.

Logging

Irrespective of the application, logging is core to keeping an eye on the internals of the network communication. Developers using veilid-core do not use veilid server or its config file directly, but they set the config through the VeilidConfig object during api startup. Logging is done separately. Each language has different logging bindings.

Rust

For rust it is done through a Tracing layer. At the rust level instantiate a veilid_core::ApiTracingLayer add it to the Layer stack. There's log 'facilities' you can enable and disable, and a default list of facilities that are ignored. These are the facilities for which logs are not emitted by default and you'd have to 'ignore' those log targets (log target is the same as 'facility', terminology-wise).

pub static DEFAULT_LOG_FACILITIES_IGNORE_LIST: [&str; 29] = [
    "mio",
    "h2",
    "hyper",
    "tower",
    "tonic",
    "tokio",
    "runtime",
    "tokio_util",
    "want",
    "serial_test",
    "async_std",
    "async_io",
    "polling",
    "rustls",
    "async_tungstenite",
    "tungstenite",
    "netlink_proto",
    "netlink_sys",
    "hickory_resolver",
    "hickory_proto",
    "attohttpc",
    "ws_stream_wasm",
    "keyvaluedb_web",
    "veilid_api",
    "network_result",
    "dht",
    "fanout",
    "receipt",
    "rpc_message",
];

These are the enabled facilities:

pub static DEFAULT_LOG_FACILITIES_ENABLED_LIST: [&str; 8] = [
    "net",
    "rpc",
    "rtab",
    "stor",
    "client_api",
    "pstore",
    "tstore",
    "crypto",
];

Dart

For Dart programs, with FFI, you instantiate a VeilidFFIConfig object with your logging config. Here is an example from veilid-flutter/example/lib/veilid_init.dart in the example code for veilid-flutter:

// Initialize Veilid
// Call only once.
void veilidInit() {
  if (kIsWeb) {
    const platformConfig = VeilidWASMConfig(
        logging: VeilidWASMConfigLogging(
            performance: VeilidWASMConfigLoggingPerformance(
                enabled: true,
                level: VeilidConfigLogLevel.debug,
                logsInTimings: true,
                logsInConsole: true),
            api: VeilidWASMConfigLoggingApi(
                enabled: true, level: VeilidConfigLogLevel.info)));
    Veilid.instance.initializeVeilidCore(platformConfig.toJson());
  } else {
    const platformConfig = VeilidFFIConfig(
        logging: VeilidFFIConfigLogging(
            terminal: VeilidFFIConfigLoggingTerminal(
              enabled: false,
              level: VeilidConfigLogLevel.debug,
            ),
            otlp: VeilidFFIConfigLoggingOtlp(
                enabled: false,
                level: VeilidConfigLogLevel.trace,
                grpcEndpoint: 'localhost:4317',
                serviceName: 'VeilidExample'),
            api: VeilidFFIConfigLoggingApi(
                enabled: true, level: VeilidConfigLogLevel.info),
            flame: VeilidFFIConfigLoggingFlame(enabled: false, path: '')));
    Veilid.instance.initializeVeilidCore(platformConfig.toJson());
  }
}

Python

Python doesn't have FFI or WASM support yet, and uses the JSON API only right now so that required veilid-server, and just uses the veilid-server.conf. That's easy because you can just edit the file!

logging:
  system:
    enabled: true
    level: info
  terminal:
    enabled: false

Core

Routing Contexts

A RoutingContext is how a node talks to other nodes.

Once a Veilid node is attached, it receives the fundamental routing information for the network. However, this doesn't allow for anonymity for either a sender or receiver, or for other communications preferences to be set. As such, a RoutingContext needs to be created.

A RoutingContext can be either "direct" or "with safety". If safety is selected, it will route messages via a safety route.

All RoutingContexts have a variable preference for Sequencing: Ensure ordered messaging, prefer unordered messaging, or have no preference about ordered or unordered messaging. The default is to have no preference. In the "direct" RoutingContext case, there are no other attributes which can be or need to be set.

A RoutingContext with safety has 3 additional attributes:

1. An optional preferred beginning for the route, in the form of a RouteID.
2. Hop count: How many extra hops to create the route for. It must be greater than 0, and the default is 1.
3. Stability: Prefer either low latency or long uptime. The default is to prefer low latency.

It is possible to have multiple RoutingContext objects simultaneously, with different preferences.

DHT Schema

In Veilid, the DHT (Distributed Hash Table) is a set of "records" which are stored across nodes on the network. A record has a "lookup key" and a "value". The lookup key is used to look up (and potentially retrieve from other nodes) the value of the record. The value is composed of a "schema" and a list of "subkey values", where each subkey has an ordinal index (0, 1, 2, etc).

The schema is a declaration of who can store information under a given sub-key, and is used to validate attempts to update or write to those subkey values. Depending on the schema form used, the creator of the DHT record (the 'owner') or one of a selected set of 'writers' listed in the record's schema will be allowed to write to a given subkey.

There are (as of 0.2.6) two types of schema. One, tagged "default" or "DFLT", allows only the keypair of the creator of the DHT entry to store information in it or its subkeys.

The other type of schema was created for a simple form of a social media, and is tagged "simple" or "SMPL". It allows the schema creator to assign (at creation time) permission to write to other subkeys to other signing keypairs.

Once created, the schema cannot be changed, because it is hashed to form part of the lookup key. Any alterations would change the hash, and thus also change the name of the DHT record.

Default (DFLT)

Simple (SMPL)

AppCall and AppMessage

If you want to message an app directly via RPC, without necessarily having to wait for the DHT to make your changes available to your target, AppCall and AppMessage are where you should look.

It is very important to recognize this: Veilid does no interpretation or checking of the bytes in the AppCallQ, AppCallA, and AppMessage packets. You can truly shoot yourself in the foot here by accepting events for calls and messages which have not been authorized in some manner within your application. How you do so is entirely up to you. (Ideally, when the Veilid Standard Library comes out, it will have one or several mechanisms you will be able to use.)

AppCall is a query to a remote application, with an expected reply. AppMessage is a statement to a remote application, without an expected reply. Or, put another way, AppMessage is a call to a function that doesn't return anything.

Replying to an AppCall

To reply to an AppCall, the method VeilidAPI::app_call_reply can be used. This uses the OperationID member of the AppCall event.

Sending an AppCall or AppMessage

To send an AppCall or AppMessage, you must have a RoutingContext, typically obtained via a call to routing_context() or by cloning an existing RoutingContext.

You must also have a target NodeID.

Callback Messages

The update callback function receives ALL of the events that are relevant to the Veilid network and the local node. It is the application's job to filter them and decide what to respond to. This callback will receive messages of the following forms.

The various update types

Log

Log

This is a log message from the Veilid API. If the configuration does not specify the log level, it defaults to Error. Only messages with a value less than or equal to the configuration's log level will be sent. It has the following members:

• level: 1 byte which expresses how important the message is (lower values are more important), with the following meanings:

• message: The log string.
• backtrace: An optional backtrace (in the veilid-core code) of where this log line was created.

AppMessage

AppMessage

This is a message sent to the application running on this node, with no expectation of a response.

Veilid does no interpretation or modification of the AppMessage bytes sent.

It has the following members:

• sender: An optional node public key for what node sent this message.
• message: A Vec<u8> containing the message as sent by the sender.

AppCall

AppCall

This is a remote procedure call to a procedure which the sender thinks is hosted by this application, with a response solicited. The receiver can safely ignore this message if it does not want to reply.

Veilid does no interpretation or modification of the AppCall bytes sent.

It has the following members:

• sender: An optional node public key for what node made this procedure call.
• message: A Vec<u8> containing the call and parameters as sent by the sender.
• OperationID: A value which must be passed to app_call_reply to indicate which appcall is being responded to.

If an AppCall is not responded to, it will time out, and the sender must choose whether or not to re-send it. The local Veilid node tracks OperationIDs that have not yet timed out, and will clean them up if they're not responded to.

Attachment

Attachment

Regardless of what other systems refer to as "attachments", this is a notification of how well this Veilid node is attached to the network. Conceptually, it provides a "signal bar" mechanism for ease of understanding, based on the number of recent peers it has seen. The numbers given in the table are defaults, and can be set in the node configuration file.

The Attachment message has the following members:

• state: One of the above values, describing how well the node is currently attached to the Veilid network.
• public_internet_ready: A bool stating whether we are attached over the public internet, and if our network class and dial info have been determined.
• local_network_ready: A bool stating whether we are listening on the local network.

Network

Network

This message conveys changes in the node's state, regarding network connectivity. It provides statistics for the whole node, not just any particular network interface that the node may be using.

This message has the following members:

• started: A boolean which indicates if the network has started.
• bps_down: The bytes per second (bps) which have been received.
• bps_up: The bytes per second (bps) which have been sent.
• peers: A Vec<PeerTableData> which contains a list of all recently connected peers.

Because PeerTableData is a complicated structure, it will be discussed later. [[ Note to self: WRITE ABOUT IT ]]

Config

Config

This message conveys changes in the node's states, regarding its running configuration. It provides a single member, which contains a copy of the running configuration.

• config: A VeilidConfigInner object which provides information about the new current configuration.

Because VeilidConfigInner is a complicated structure, it will be discussed later. [[ Note to self: WRITE ABOUT IT ]]

RouteChange

RouteChange

This message is sent when dead routes are detected on the network. It contains the following members:

• dead_routes: A Vec<RouteId> of locally allocated routes (safety routes and allocated private routes) which are no longer directly accessible. These have already been deallocated by the time this message is sent.
• dead_remote_routes: A Vec<RouteId> of routes which are no longer accessible on the network. This may include private routes which have been received from other nodes.

ValueChange

ValueChange

(As of v0.2.4, this message is defined, but the functionality to request it is not available.)

When a DHT entry that this node has set a Watch on is updated, this message is sent.

The DHT is a topic of its own, and will be discussed in its own section. [[ Note to self: WRITE ABOUT IT ]]

This message contains the following members:

• key: The key of the DHT entry
• subkeys: A Vec<ValueSubkey> of the DHT entries that were updated.
• count: The number of updated DHT subkeys in the subkeys vector.
• value: A ValueData structure containing the update sequence number, the value, and the writer.

Shutdown

Shutdown

When the node is shutting down and will no longer accept requests or send updates, this message will be sent. It has no members.

API Reference

The Rust API reference can be found at https://docs.rs/crate/veilid-core/.

(links to pydoc)

(links to flutter doc)

Rust API

The rustdoc for the veilid-core crate will always be more authoritative than this book. If you have specific improvements for the rustdoc, please open a ticket at the Veilid issue tracker.

Rust veilid-core crate functions

The veilid-core crate exposes a few functions that aren't attached to any particular type. These functions are generally helpful tools to work with common kinds of data one expects Veilid to work with, and two functions related to starting up the Veilid node.

VeilidAPI object

The VeilidAPI object is the root of all interaction with the Veilid node instance which the API accesses. To obtain the VeilidAPI object, call either api_startup or api_startup_json For more information, see the Instantiating Veilid section.

From [VeilidAPI], the following can be accessed:

In addition, the following functions can be performed:

• Reply to AppCall RPCs
• Attach and detach from the network
• Create and import private routes

VeilidAPI functional surface

VeilidAPI exposes the following functions:

pub async fn shutdown(self) pub fn is_shutdown(&self) -> bool pub fn config(&self) -> VeilidAPIResult pub fn crypto(&self) -> VeilidAPIResult pub fn table_store(&self) -> VeilidAPIResult pub fn protected_store(&self) -> VeilidAPIResult pub async fn get_state(&self) -> VeilidAPIResult pub async fn attach(&self) -> VeilidAPIResult<()> pub async fn detach(&self) -> VeilidAPIResult<()> pub fn routing_context(&self) -> RoutingContext pub async fn parse_as_target<S: AsRef>(&self, s: S) -> VeilidAPIResult pub async fn new_private_route(&self) -> VeilidAPIResult<(RouteId, Vec)> pub async fn new_custom_private_route(&self, crypto_kinds: &[CryptoKind], stability: Stability, sequencing: Sequencing,) pub fn import_remote_private_route(&self, blob: Vec) -> VeilidAPIResult pub fn release_private_route(&self, route_id: RouteId) -> VeilidAPIResult<()> pub async fn app_call_reply(&self, call_id: OperationId, message: Vec,)

VeilidConfig

Crypto

TableStore

ProtectedStore

RoutingContext

A RoutingContext is how a node talks to other nodes.

Once a Veilid node is attached, it receives the fundamental routing information for the network. However, this doesn't allow for anonymity for either a sender or receiver, or for other communications preferences to be set. As such, a RoutingContext needs to be created.

A RoutingContext contains three things:

Veilid Flutter Example

To set up a new Veilid enabled Flutter project, you can start by grabbing the veilid-flutter directory out of the main project, and putting it right alongside the Veilid directory in your projects folder. You can also start with the VeilidChat application since it is made with Flutter as well.

Flutter projects at 10,000 Feet

Get in the blimp, we are looking at this from way above.

1. Clone veilid
2. Clone veilid-chat
3. Run veilid/dev-setup/install_linux_prerequisites.sh
4. Run veilid/dev-setup/setup_linux.sh
5. Run veilid-chat/dev-setup.sh
6. Run veilid-chat/build.sh
7. Install your development environment and install its flutter plugin
8. Open veilid-chat project in your IDE
9. Adjust the configuration between Veilid, Flutter, and the operating system as needed
10. Build and deploy

Step by step writeups for different environments

• Using Android Studio in Debian
• Working in Linux on the command line
• Building Windows apps

Building Veilid Flutter apps for Android

Building Veilid Flutter apps for Linux

Building Veilid Flutter apps on Windows

For Windows, starting with the VeilidChat application to structure your own application is a really fine idea. There are tricky build considerations that have already been worked through, and reusing that code is a good thing.

The first thing you need is the Veilid and VeilidChat source code, from Gitlab. Open an administrative command prompt, navigate to your default coding folder (I use c:\\code, YMMV), and clone like crazy:

git clone https://gitlab.com/veilid/veilid.git

git clone https://gitlab.com/veilid/veilidchat.git

Getting set up

There are more than a few bits you might need to add to your machine. If you are starting relatively from scratch, you'll need all of this. If you already have Rust or Flutter, then you can obviously skip those.

CapnProto is a binary encoding library, and provides some features needed to translate between Rust and C++ and Dart. I know it seems like a joke application on the site, but it is serious software.

https://capnproto.org/install.html

Speaking of Rust, Veilid is written in Rust, we might need that.

https://www.rust-lang.org/tools/install

Building VeilidChat

Not it's time to run some scripts. In the administrative command prompt launched above, navigate to /veilid and run:

dev-tools/setup.bat

Assuming no errors, switch over to /veilidchat and get started there. First, though, you will need Flutter.

One of the things that need to happen here is installation of Visual Studio 2022 Community Edition from the link on Flutter's install instructions. This is different from VScode, a common confusion. Microsoft is not so good at naming things. Make sure you install with the "Desktop Development with C++ Workload."

https://docs.flutter.dev/get-started/install/windows/desktop

Flutter isn't as much of an installer as it is an "unpack and edit the path" kind of thing. I followed the instructions on the Flutter page, and you should too because they are subject to change.

Note that you will need CMD not PS terminal to run flutter. From a prompt, run:

flutter install windows --release

And select 1 for Windows.

This will clean set up for the full build., which is:

reset_run.bat

It will ask once again if you want to work with Windows or another installed platform. for this, again, select Windows. Then, we wait.

Using VSCode

Coming soon! We wanted to get the above out sooner rather than later.

Veilid Python Chat

This chat program is meant to be a simple, readable demonstration of how to communicate with Veilid. A real chat program would not look like this:

• A lot of code is duplicated. This was on purpose so that each function would be readable and understandable without having to trace through lots of helper functions.
• The user interface is too simple. If you're halfway through typing a message when your app receives one from the other party, the inbound message will make a mess of your screen.
• The messaging flow is too simple. If you send messages faster than the other party can receive them, some of your messages old will get overwritten by new ones.
• Users have to trade their public keys, and even the chat's DHT key, through another channel like SMS.

But then, the major portions of a real app wouldn't fit on a single screen. This demo just shows how all the moving parts fit together. Your challenge is to take them and build something better.

To take you further down that path, there are details in this section, as follows:

1. Getting installed
2. Using python-chat
3. [internal details](An overview on how it is put together)

Installation

1. The python chat application requires a headless Veilid node. Find instructions here.
2. Clone the python-demo project.
3. Install poetry if you haven't already.
4. Run poetry install

Usage

First, run veilid-server. This demo tries to connect to localhost port 5959 by default. You can override that with the --host and --port arguments.

Create your cryptographic keypair:

$ poetry run chat keygen
Your new public key is: d3aDb3ef
Share it with your friends!

This writes your new private key to the local keystore. Do not store your local keys to other files, they are not encrypted!

Copy the public key and send it to a friend you want to chat with. Have your friend do the same.

Now, add your friend's public key to your keyring:

$ poetry run chat add-friend MyFriend L0nGkEyStR1ng

To start a chat with that friend:

$ poetry run chat start MyFriend
New chat key: VLD0:abcd1234
Give that to your friend!
SEND>

Copy that chat key and send it to your chat partner. They can respond to your chat:

$ poetry run chat respond CoolBuddy VLD0:abcd1234
SEND>

Now you can send simple text messages back and forth. Your messages are encrypted and transmitted through a distributed hash table (DHT), passed through an open-source, peer-to-peer, mobile-first networked application framework. Neat!

Remember that this simplified program can only receive a message when it's not waiting for you to enter one.

Sequence diagram

Here's how the program generates keys, uses the Diffie-Hellman (DH) algorithm to calculate shared keys, reads from and writes to the DHT, and uses encryption with nonces to secure messages in the public network.

sequenceDiagram
    # Parties involved:
    actor Alice
    participant Va as Alice's Veilid
    participant Magic
    participant Vb as Bob's Veilid
    actor Bob


    Note over Alice,Bob: Alice and Bob generate keypairs.
    Alice ->> Va: Generate key
    Va ->> Alice: keypair
    Bob ->> Vb: Generate key
    Vb ->> Bob: keypair
    Note over Alice,Bob: Alice and Bob send their public keys to each other.
    Alice -->> Bob: Alice's pubkey (out-of-band)
    Bob -->> Alice: Bob's pubkey (out-of-band)

    Note over Alice,Va: Alice starts a chat by<br/>getting a shared secret.
    Alice ->> Va: cached_dh()<br>(Bob's pubkey, Alice's secret key)
    Va ->> Alice: secret

    Note over Alice,Va: Alice creates a DHT record.
    Alice ->> Va: Create DHT record
    Va ->> Alice: DHT key

    Note over Alice,Bob: Alice sends the DHT record's key to Bob.
    Alice -->> Bob: DHT key (out-of-band)

    Note over Bob,Vb: Bob creates a shared secret.<br/>It will be the same as Alice's.
    Bob ->> Vb: cached_dh()<br>(Alice's pubkey, Bob's secret key)
    Vb ->> Bob: secret

    loop Until done
        Note over Alice,Bob: Alice sends a message to Bob.

        Note over Alice,Va: First, she creates a random nonce.<br/>This is done for each message.
        Alice ->> Va: random_nonce()
        Va ->> Alice: nonce

        Note over Alice,Va: Then she encrypts the message with<br/>the shared key and the nonce.
        Alice ->> Va: encrypt("Message", secret, nonce)
        Va ->> Alice: ciphertext

        Note over Alice,Va: Alice updates the DHT record's "0" subkey<br/>with the nonce plus the encrypted text.
        Alice ->> Va: set_dht_value(0, nonce+ciphertext)

        Note over Va,Vb:  Veilid magic happens!
        Va ->> Magic: Updated DHT key
        Magic ->> Vb: Updated DHT key

        Note over Bob,Vb:  Meanwhile, Bob is polling the DHT key's "0"<br/>subkey, waiting for a message to appear.
        Bob ->> Vb: get_dht_value(0)
        Vb ->> Bob: nonce+ciphertext

        Note over Bob,Vb: Bob decrypts Alice's encrypted message<br/>with the shared secret and the nonce<br/>that was send with the ciphertext.
        Bob ->> Vb: decrypt(ciphertext, secret, nonce)
        Vb->> Bob: "Message"

        Note over Alice,Bob: Bob replies to Alice with a similar process.

        Note over Bob,Vb: First, he asks for a random nonce.<br/>Again, he uses a different nonce each time!
        Bob ->> Vb: random_nonce()
        Vb ->> Bob: nonce

        Note over Bob,Vb: Bob encrypt's his reply with<br/>the shared secret and nonce.
        Bob ->> Vb: encrypt("Reply", secret, nonce)
        Vb ->> Bob: ciphertext

        Note over Bob,Vb:  Although Bob reads from subkey 0,<br/>he writes to subkey 1.
        Bob ->> Vb: set_dht_value(1, nonce+ciphertext)

        Note over Va,Vb: More Veilid magic!
        Vb ->> Magic: Updated DHT key
        Magic ->> Va: Updated DHT key

        Note over Alice,Va: Alice is polling subkey 1 for Alice's<br/>encrypted response.
        Alice ->> Va: get_dht_value(1)
        Va ->> Alice: nonce+ciphertext

        Note over Alice,Va:  Alice decrypts Bob's encrypted reply<br/>with the shared secret and the<br/>nonce accompanying the message.
        Alice ->> Va: decrypt(ciphertext, secret, nonce)
        Va ->> Alice: "Reply"

        Note over Alice,Bob: ...and the repeat until one of them closes the chat.
    end

VeilidChat

Project Layout

How VeilidChat Uses Veilid

WASM Example

Glossary Of Terms

Attachment: The measure of how many connections a node currently has to the Veilid network.

API: Application programming interface. Basically a shared language which allows two or more programs to talk to each other. An API is an agreed-upon format for how requests and responses between them will be shaped, and what information they must contain.

Bootstrap: To initialize a program without prior state data.

Callback function: A function which the application developer writes and which modifies the application state, which is provided to the Veilid library for it to call when network events happen.

Cargo: see “package manager”.

Crate: A single, compilable unit of code in Rust.

Creator keypair: The keypair owned by the creator of a DHT record.

Cryptography: The process of obscuring information (“encrypting”), so that it can only be understood by the intended recipient.

Default schema: A DHT schema type which allows only the creator keypair to write to its value or any of its subkey values.

DNS: Domain name system. A naming scheme for things on the internet, which among other things matches hostnames that humans can easily remember (like google.com) to their corresponding IP addresses (like 172.217.16.238).

DHT: Distributed Hash Table, a key-value store which stores its keys and values across many nodes, and which provides a means to look them up. In Veilid, every DHT record has a key, a schema, a creator keypair, and subkeys with values. If it uses the Simple schema, it may also have writer keypairs for one or more of its subkeys.

Dialinfo: The collection of a node's public key, location on the physical network, and physical protocols it can accept connections on.

Encryption: see “Cryptography”.

Headless: A headless program runs without a graphical user interface, for example via command line. In the context of Veilid, a node can also be headless; it runs without an application, and its only function is to move traffic.

IP Address: A string of numbers assigned to each device that connects to the internet, which allows it to be uniquely identified.

Internet: a series of tubes.

Keypair: A set of two cryptographic keys. The public key, which can be freely shared, is for encrypting plaintext into ciphertext. The private one is for decrpyting, and must be kept secret.

Node: Any single point on a network. In Veilid, a node is an instance of the veilid-core code run by an application. (for properties of a node, see 2.1. Nodes).

Package manager: software which automatically installs, configures, and removes programs in bulk. Package managers reduce repetition, reduce the odds of a human making a mistake, and ensure each program comes from a trusted source. The package manager that Rust (and therefore Veilid) uses is called “Cargo”.

Repository: The place where your software packages are stored. Repo for short. For Rust projects, this is crates.io

Routing: the process of selecting a path for traffic in a network; which nodes will receive individual network packets on their way to their end destination.

Rust: The programming language that Veilid is built in. To learn more about what makes Rust unique, see their documentation here. (https://www.rust-lang.org/)

Safety route: A route designated by a sending node to protect its physical network connection.

Secret route: A route designated by a receiving node to protect its physical network connection.

Simple schema: A DHT schema type which allows the creator keypair to delegate the ability to write to subkeys to some other keypair or keypairs.

Topology: the structure of a computer network; it’s a description of how data is sent from one device to another. A network topology could be centralized around a central authority, decentralized/federated between many, or distributed across an entire peer-to-peer community.

Tunneling: Moving data from one network to another.

Veilid-core: The main library used to create a Veilid node and operate it as part of an application.

Writer keypair: A keypair owned by someone the creator of a DHT record has given the ability to write to a subkey or several subkeys.

About the Veilid Foundation

The Veilid name and brand are owned by the Veilid Foundation, a nonprofit that is dedicated to making sure that apps and networks built using the Veilid framework match the ideals of this project. We will have a public grading system on public projects using Veilid upon things like accessibility, transparency (open source), and non monetization. We will speak out publicly about any immoral uses of the framework, and will use the Veilid name to promote and support those who stick with the ideals we present. The foundation board is made up of:

• Christien “dildog” Rioux
• Katelyn “medus4” Bowden
• Paul “Gibson” Miller

The Foundation exists to support the entire project and provide assistance to those building projects within the core ideals spelled out previously, along with making sure that anything built using the Veilid community is legal, safe, and accessible to all. The Veilid Foundation does not support or encourage anyone using the framework for illegal purposes.

The Veilid Foundation, Inc. is a 501(c)3 corporation chartered with the state of Virginia.

We exist to develop, distribute, and maintain a privacy focused communication platform and protocol for the purposes of defending human and civil rights.

We can receive postal mail at:

Veilid Foundation, Inc. P.O. Box 1917 Leesburg, VA 20177

Donations (tax-deductible in United States) can be made via Stripe, our payment processor, at https://veilid.org/donate/.

For more information, please visit https://veilid.org/about-us/.