Farcaster Hubs

How do you develop on top of the Farcaster protocol? Dive in. Focus is on giving you just as much documentation as needed as well as conceptual overviews and examples. We link out to detailed project documentation where used.

What is a Farcaster Hub?

Farcaster works by some servers storing history of the social network and one being able to add new history (e.g. posts, likes, follow, …) to them.

These servers are called hubs. If you are familiar with blockchains, you may think of them as nodes.

Hubs verify that the rules of the social protocol are followed. How the social protocol differs from the social network you ask? The social network is the content and relationships between accounts. Say I post something, you follow me or maybe someone likes a post that is part of the social network.

But to verify that only you as the owner of the account or applications that you authorize may post is defined in the social protocol. To verify these rules and run the underlying social protocol that the social network is built on top of we utilize Hubs.

A hub is software running on a server to verify the rules of the social protocol while also storing the data of the social network.

Many people and companies run hubs, which creates a distributed network of different parties verifying the social protocol rules are adhered to. Anyone can run a hub and be part of verifying rules as well as storing or adding data.

Where do Hubs fit within the greater Farcaster architecture

Hubs are the base layer, the ground truth. Besides identity (e.g. your account - FID) which is stored and verified on the blockchain (Optimism Mainnet), Hubs store and verify everything going on in the Farcaster protocol.

Everything we build on top of Farcaster relies on people running hubs and hubs being available.

You build APIs on top of data from Hubs being replicated and enriched into databases, that then serve the API queries.

You use Hubs to verify signatures and make sure that a message you see is valid.

For a message to be accepted, the fid must be registered in the Id registry, and signed with a valid signer present the Key registry, and the fid must have enough storage allocated in the Storage registry.

As stated they are the source of truth.

The Details about Farcaster Hubs

One thing people do not realise is that not all data is on hubs, but they also miss that all data on Hubs is public. This is because the data on hubs is the social network data, and the social network is public.

Though some client or app data is only available in that client/app. An example of that is views. If you view a post there is no message sent on Hubs that you viewed it, but the client knows. That is why some clients will show you views, which is an underestimate as people are seeing your posts across all clients.

The data that is on hubs is described in the Farcaster protocol specification.

The following is an overview of all details regarding Farcaster hubs, answering the most frequently asked questions with links to implementation and references where viable.

Where does a Farcaster Hub run?

Since a Farcaster Hub is just software it can run anywhere you want to run your software. This can be your phone, your home computer, laptop or a server. Hub operators may also decide to run Hubs on AWS, or another cloud service like Google Cloud or Azure. Many developers run their hubs on a VPS from Hetzner or others.

In the end there are many ways of running a Hub in Kubernetes, Docker or straight from source. The Software can be run anywhere.

Who is running Farcaster Hubs?

While anyone who wants to run the Hub software can run Hubs many Hub operators are enthusiasts, who want to own their data or explore the data through hobby projects. Commercial operators are companies, who built their product offerings on top of Farcaster data and therefor want the control of their data and lowest latency.

How does running a Hub support the network?

Running a Hub supports the network by verifying the rules of the social protocol and storing the data of the social network. This is important as it is the base layer of the Farcaster protocol. Anyone replicating data makes it more fault tolerant. Anyone verifying the rules of the network are upheld, makes it more resistant and fault tolerant. Additionally running your own hub for your app means you have the lowest latency and can serve users better, which in turn provides Farcaster users are better experience improving the overall network user experience.

What are the rewards for running a Farcaser Hub?

While there are no direct benefits or rewards for running a Farcaster Hub you gain a copy of all data and a short path to access Farcaster.

The low latency can mean that as a developer your software runs way faster leading to a better user experience. For example a Farcaster bot needs to respond as fast as possible. It helps if that bot runs of a hub directly since that likely offers the lowest latency.

What does a Farcaster Hub do?

From the official docs “Hubble creates a private instance of Farcaster on your machine. It peers with other instances and downloads a copy of the entire network. Messages uploaded to your Hubble instance will be broadcast to the network.”

Which translates to it talks to other Hubs in the network to make sure your local data copy is up to date and you have the ability to add your own data (e.g. comments or posts, changing profile picture and more). Data meaning Messages, which are the base layer of the Farcaster protocol. Learn more about Messages

How do Farcaster Hubs send Messages?

Messages are exchanged between Hubs using libp2p under the hood. This leverages a wide spread opensource library for peer to peer networks.

Meaning that each Hub that is used to write to the Network (recieve messages and push these new messages to the network) tells a set of other hubs (it’s peers) about this. Then these talk to their peers and that way the messages propagate until everyone is eventually in sync.

That peer set can be seen as your friend group and how many peers (friends) you have is called peer count. Say you want to tell each friend the name of the cool pokemon you discovered. You have 4 friends that you have direct contact with (peer count 4, these 4 friends make up your peer set). When you tell these friends they tell their peers (their close friends) and after all do it, everyone in your larger friend group will know. That’s how this sync process works.

To learn more about libp2p check their webpage and to learn how it is used in hubs the Hubble codebase contains the Farcaster implementation.

How do Farcaster Hubs recieve Messages?

Recieving messages is done through a number of ways, while the Hub is running it’s done by checking in with peers (other Hubs) to make sure you have all messages and if not, recieving missing messages from them.

If you’re just starting your Hub though, then you can opt into snapshot synchronization where you choose a peer that offers snapshots and then start with that already. This means that you are getting all the messages this hub has at the time of the snapshot. With a Snapshot you do not need to go back in history as much as without it until you are fully synchronized. The Warpcast team (Merkle Manufactury) publishes a Snapshot every day 9:00AM PT, which is used as default Snapshot for sync.

New Messages are propagated through peer to peer networking as when sending messages

How do Farcaster Hubs store Messages?

All messages when using Hubble are stored using RocksDB. RocksDB is a key-value store that is optimized for fast storage and retrieval of data.

Your hub recieves messages and adds them to RocksDB if they are new and that way builds up a store of all messages. Additionally many folks choose to write these messages to their own database, but as far as hubs themselves are concerned the Hubble implementation uses RocksDB for storage.

Where can I find these stored Farcaster Messages on Hubs?

The RocksDB storage for messages on Hubs can be found in the .rocks directory which when using the Docker Image is present at /home/node/app/apps/hubble/.rocks.

How do Farcaster Hubs synchronize?

Farcaster Hubs synchronize by recieving messages from other Hubs (their peers) and by sending their own new messages to other hubs (their peers) and as everyone sends to their peers eventually the whole network will have recieved the messages and be synchronized. An overview with practical example is available here.

The technical details are covered in the how do Farcaster hubs works section.

What part of Farcaster Hubs is onchain?

Farcaster is not completely onchain, though Farcaster identities are. The usernames are onchain and the amount of messages the users are allowed to consume in terms of storage on Hubs is. Meaning user identity is managed onchain, while data storage is offchain on Hubs. The Messages that Hubs exchange are stored on Hubs as explained here.

Which Chains is Farcaster on?

There are multiple onchain parts of farcaster. The Signer (Key) Registration, the storage registration and identity as well as the username (fname) registry.

All of these are on Optimism Mainnet except for the fname registry, which is on Ethereum Mainnet.

The reason for Ethereum Mainnet on the fname registry is, that it is ENS based. The Key (Signer) Registration happens on Optimism Mainnet via the Key Registry.

The Storage Registration happens on Optimism Mainnet via the Storage Registry.

To see a full live overview of the contract addresses and where each part is deployed refer to the contracts repo.

To learn more about the full architecture, you can read and watch the full Farcaster architecture breakdown.

How to read Farcaster Onchain Data?

The easiest way to read Farcaster’s onchain data is to use Hubs. The Hub API stores onchain events as messages, just like other Farcaster events and you can query them. Though one can also look at the blockchain directly and watch the Smart Contracts. Popular tools to do so are viem and ethersjs.

How to use Farcaster Smart Contracts?

If you want to write to the Smart Contracts you can read the Code on Github or the Reference in the Farcaster docs. What you can do with the Smart Contracts is register a new FID (user account) and extend or add storage so you can post, react or follow more. Additionally since Signers are managed via Smart Contracts, you can also let users register public keys that can be used to act on their behalf. This is useful for writing in the name of a user if you’re a content scheduling platform for example.

What is Farcaster Storage?

Farcaster Storage refers the amount of messages (likes, follows, casts, …) that one account is able to have stored on Hubs. In the event of exceeding one’s storage limit the oldest messages will be deleted from Hubs and therefore not be available anymore. Yes this means your old posts may no longer be available or your likes that you gave could be gone.

How does Farcaster Storage work?

Every Hub checks the blockchain’s smart contracts to verify the storage limit of every account. If an acount goes over that limit, the Hub will not retain old messages. Think of it this way: The Hub is an apple farmer and looking at how many apples he can store. If new Apples come in, which are above his available storage capacity, then he wants to store only the newest apples. To make room for the new apples, he removes the oldest apple each time.

Though Hubs do not delete the messages instantly, there is a grace period in which one can buy/register new storage capacity with the smart contracts. In case this happens, the updated storage limits will be applied so if extended, the messages will not be deleted.

Not every message is treated equal as the storage limit is split into different limits. Posts (casts, replies, quote casts) are tracked individually as well as reactions (likes) and links (follows). This means that if the storage limit for reactions is exceeded there will NOT be a deletion of your old posts, but of old reactions.

In the apple farmers example this means he has three storage facilities for different kinds of apples. He does not throw away and replace apples that do not belong into the facility that is over capacity. Why? Because these can be stored still in accordance to the limits.

To summarize from a user perspective if one goes beyond their storage limits (which are stored on the blockchain), one can extend storage by paying for more at the smart contracts. In case this happens within the grace period, the updated limits will be applied. If not for any new message (like, reaction, etc) that is of the same type that is beyond the limit, the oldest messages will be deleted (pruned).

If you want to check your storage caapcity, you can do so from within your Farcaster client (like Warpcast) or use Fraems or tools like caststorage.com

How do Farcaster clients use Hubs?

What do clients do with Hubs?

Farcaster clients use Hubs as their source of truth. The Hubs store the networks content. As clients want to display that state and change it, they eventually read and write from Hubs. That is why for a Farcaster client Hubs are needed. Though as not everyone wants to run Hubs themselves, API Providers exist to run Hubs for you.

In the case of clients not running their own hubs, the API Provider will either just run a Hub for them and the client then uses the managed hubs or the provider will expose a full blown API based on Hub data. That API can be optimized for the use cases of the API providers customers. These optimizations could be aggregating data, so that there is less computation on the API consumer or enriching the data, combining data and more.

Why do Farcaster clients run Hubs?

When operating a Hub the client developers get direct access to the underlying Farcaster social graph (data). By running a hub themselves, the latency for writing to Hubs is significantly reduced and there is no dependency on third party Farcaster APIs. It also allows direct event streaming when reading data, so that actions can be taken as fast as possible based on events within the Farcaster network.

Does a Farcaster client need a Hub?

Technically every client needs a hub, the question being who is running it. Farcaster data is stored on hubs, so any write or read eventually reaches Hubs, though if the client developers do not operate their own hub directly, then a 3rd party will. In summary this means, yes every client needs a hub, but the client team may not operate the hub themselves.

Why do Farcaster clients need a database?

Hubble (the Hub implementation) stores data by holding a record of all messages sent. This requires everyone who wants to show the number of likes (reactions with type like) for a post to perform a lookup on all messages counting those where this post has been liked and then substracting those where the like reaction has been removed. Since these queries are not efficient and storing the aggregate in connection to the post may be preferred in addition to the full message history or even a list of everyone who liked the post, one needs a custom data storage solution. This is where storing the data in one’s own database comes to the rescue. You may also already have data that you need to keep like payment status in case your users have to pay for service and other data, which means one likely has an application database anyways.

How to post to the Farcaster Database?

If you need to ingest Farcaster data into your database, you can subscribe the a Hubs gRPC Event Stream and add everything you observe to the database. Doing so provides you with your own history of all messages currently new on the Farcaster network. To get historic data you can backfill from the data stored on a hub.

Developing a solution to write from Event Stream to the database is rather straight forward since the data format is well defined. Additionally the Merkle Manufactory team provides a ready to use indexing solution in the hub-monorepo on Github. When writing to the database one could also add custom logic to perform aggregates compution, trigger webhooks or anything else.

Recall that your database is not a Farcaster hub! If you want to write to the database you’ll have to do so by writing your state changes to Farcaster hubs. Here’s how you can write to hubs. Working with your database, will be specific to the database solution you chose.

How to read Farcaster Data from your Database?

If you have data in your database, then quering it is the same as for any other call to this database. If one is using PostgreSQL then, you’ll be using SQL queries. If one is using another database, then this will have it’s way of reading (querying) data.

How to query Hubs?

Hubble (the main hub implementation) provides to APIs to query data: a GRPC and a HTTP API. Depending on your needs you may combine them or use one of them.

The Info endpoint of the HTTP API is useful to check readyness and see when a hub is up and can verify messages. The hub being online/ready may not mean that it has fully synchronised. One needs to compare to the network to see the sync percentage of a hub.

Querying Hubs for data then is calling these APIs and doing the computations over the messages they return. Besides getting data one can use them to verify messages as well (e.g. Frame Message Validation in Frames).

How to write to Hubs?

Hubble (the main hub implementation) provides to APIs to write data: a GRPC and HTTP API. To call them and write to hubs you need to have a private key matching the public key stored for the FID you want to write as. These key pairs are called Signers. Hubs will verify that a message is signed by a valid signer and if not reject the message.

Assuming you have a signer, writing becomes constructing the message and then calling the respective API endpoints on the Hub.

The Farcaster Hub API

To learn more about the API you can read the reference for the HTTP API and GRPC API.