Messages transmitted to Solana validators must not exceed the IPv6 MTU size to ensure fast and reliable network transmission of cluster info over UDP. Solana's networking stack uses a conservative MTU size of 1280 bytes which, after accounting for headers, leaves 1232 bytes for packet data like serialized transactions.
Developers building applications on Solana must design their on-chain program interfaces within the above transaction size limit constraint. One common work-around is to store state temporarily on-chain and consume that state in later transactions. This is the approach used by the BPF loader program for deploying Solana programs.
However, this workaround doesn't work well when developers compose many on-chain programs in a single atomic transaction. With more composition comes more account inputs, each of which takes up 32 bytes. There is currently no available workaround for increasing the number of accounts used in a single transaction since each transaction must list all accounts that it needs to properly lock accounts for parallel execution. Therefore the current cap is about 35 accounts after accounting for signatures and other transaction metadata.
1) Introduce a new program which manages on-chain address lookup tables 2) Add a new transaction format which can make use of on-chain address lookup tables to efficiently load more accounts in a single transaction.
Here we describe a program-based solution to the problem, whereby a protocol developer or end-user can create collections of related addresses on-chain for concise use in a transaction's account inputs.
After addresses are stored on-chain in an address lookup table account, they may be succinctly referenced in a transaction using a 1-byte u8 index rather than a full 32-byte address. This will require a new transaction format to make use of these succinct references as well as runtime handling for looking up and loading addresses from the on-chain lookup tables.
Address lookup tables must be rent-exempt when initialized and after each time new addresses are appended. Lookup tables can either be extended from an on-chain buffered list of addresses or directly by appending addresses through instruction data. Newly appended addresses require one slot to warmup before being available to transactions for lookups.
Since transactions use a
u8 index to look up addresses, address tables can
store up to 256 addresses each. In addition to stored addresses, address table
accounts also tracks various metadata explained below.
To make it easier for address lookup tables to be updated by multi-sig or governance-controlled authorities, addresses can be buffered on-chain in a buffer account. Buffer accounts can be used to extend a lookup table with many addresses in a single small transaction.
Once an address lookup table is no longer needed, it can be deactivated and closed to have its rent balance reclaimed. Address lookup tables may not be recreated at the same address because each new lookup table must be initialized at an address derived from a recent slot.
Address lookup tables can be deactivated at any time but can continue to be used by transactions until the deactivation slot is no longer present in the slot hashes sysvar. This cool-down period ensures that in-flight transactions cannot be censored and that address lookup tables cannot be closed and recreated for the same slot.
Since address lookups require extra overhead during transaction processing, they should incur higher costs for a transaction.
In order to support address table lookups, the structure of serialized transactions must be modified. The new transaction format should not affect transaction processing in the Solana program runtime beyond the increased capacity for accounts and program invocations. Invoked programs will be unaware of which transaction format was used.
The new transaction format must be distinguished from the current transaction
format. Current transactions can fit at most 19 signatures (64-bytes each) but
the message header encodes
num_required_signatures as a
u8. Since the upper
bit of the
u8 will never be set for a valid transaction, we can enable it to
denote whether a transaction should be decoded with the versioned format or not.
- 1 extra byte for
- 1 extra byte for
- 34 extra bytes for the address and lengths of the
readonly_indexesindexes in each address table lookup
- 1 extra byte for each lookup table index
Each resolved address from an address lookup table should be stored in the transaction metadata for quick reference. This will avoid the need for clients to make multiple RPC round trips to fetch all accounts loaded by a v2 transaction. It will also make it easier to use the ledger tool to analyze account access patterns.
Fetched transaction responses will likely require a new version field to indicate to clients which transaction structure to use for deserialization. Clients using pre-existing RPC methods will receive error responses when attempting to fetch a versioned transaction which will indicate that they must upgrade.
The RPC API should also support an option for returning fully expanded transactions to abstract away the address lookup table details from downstream clients.
- Max of 256 unique accounts may be loaded by a transaction because
u8is used by compiled instructions to index into transaction message
- Address lookup tables can hold up to 256 entries because lookup table indexes are also
- Transaction signers may not be loaded through an address lookup table, the full address of each signer must be serialized in the transaction. This ensures that the performance of transaction signature checks is not affected.
- Hardware wallets will probably not be able to display details about accounts referenced through address lookup tables due to inability to verify on-chain data.
- Only single level address lookup tables can be used. Recursive lookups will not be supported.
If an address lookup table can be closed and re-initialized with new addresses, any client which is unaware of the change could inadvertently lookup unexpected addresses. To avoid this, all address lookup tables must be initialized at an address derived from a recent slot and they cannot be closed until the slot used for deactivation is no longer in the slot hashes sysvar.
Enabling more account inputs in a transaction allows for more program invocations, write-locks, and data reads / writes. Before address tables are enabled, transaction-wide compute limits and increased costs for write locks and data reads are required.
If the addresses listed within an address lookup table are mutable, front running attacks could modify which addresses are resolved for a later transaction. For this reason, address lookup tables are append-only and may only be closed if it's no longer possible to create a new lookup table at the same derived address.
Additionally, a malicious actor could try to fork the chain immediately after a new address lookup table account is added to a block. If successful, they could add a different unexpected table entry in the fork. In order to deter this attack, clients should wait for address lookup tables to be finalized before using them in a transaction. Clients may also append integrity check instructions to the transaction which verify that the correct accounts are looked up.
Address lookup table accounts may be read very frequently and will therefore be a more high profile target for denial of service attacks through write locks similar to sysvar accounts.
For this reason, special handling should be given to address lookup tables. When an address lookup table is used to lookup addresses for a transaction, it can be loaded without waiting for a read lock. To avoid race conditions, only the addresses appended in previous blocks can be used for lookups and deactivation requires a cool-down period.
Transactions may not load an account more than once whether directly through
account_keys or indirectly through
1) Account prefixes
Needing to pre-register accounts in an on-chain address lookup table is cumbersome because it adds an extra step for transaction processing. Instead, Solana transactions could use variable length address prefixes to specify accounts. These prefix shortcuts can save on data usage without needing to setup on-chain state.
However, this model requires nodes to keep a mapping of prefixes to active account addresses. Attackers can create accounts with the same prefix as a popular account to disrupt transactions.
2) Transaction builder program
Solana can provide a new on-chain program which allows "Big" transactions to be constructed on-chain by normal transactions. Once the transaction is constructed, a final "Execute" transaction can trigger a node to process the big transaction as a normal transaction without needing to fit it into an MTU sized packet.
The UX of this approach is tricky. A user could in theory sign a big transaction but it wouldn't be great if they had to use their wallet to sign multiple transactions to build that transaction that they already signed and approved. This could be a use-case for transaction relay services, though. A user could pay a relayer to construct the large pre-signed transaction on-chain for them.
In order to prevent the large transaction from being reconstructed and replayed, its message hash will need to be added to the status cache when executed.
3) Epoch account indexes
Similarly to leader schedule calculation, validators could create a global index of the most accessed accounts in the previous epoch and make that index available to transactions in the following epoch.
This approach has a downside of only updating the index at epoch boundaries which means there would be a few day delay before popular new accounts could be referenced. It also needs to be consistently generated by all validators by using some criteria like adding accounts in order by access count.
4) Address lists
Extend the transaction structure to support addresses that, when loaded, expand to a list of addresses. After expansion, all account inputs are concatenated to form a single list of account keys which can be indexed into by instructions. Address lists would likely need to be immutable to prevent attacks. They would also need to be limited in length to limit resource consumption.
This proposal can be thought of a special case of the proposed index account approach. Since the full account list would be expanded, there's no need to add additional offsets that use up the limited space in a serialized transaction. However, the expected size of an address list may need to be encoded into the transaction to aid the sanitization of account indexes. We would also need to encode how many addresses in the list should be loaded as readonly vs read-write. Lastly, special attention must be given to watch out for addresses that exist in multiple account lists.
5) Increase transaction size
Significantly larger serialized transactions have an increased likelihood of being dropped over the wire but this might not be a big issue since clients can retry transactions anyways. The only time validators need to send individual transactions over the network is when a leader forwards unprocessed transactions to the next leader.