Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement list_root_keys to the AdminKeyValueStore #3142

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1a6fc76
Insert the code for the basic stores.
MathieuDutSik Jan 15, 2025
db29208
Implement the stuff for the ScyllaDb.
MathieuDutSik Jan 15, 2025
d4f5590
Implement the feature for DynamoDb.
MathieuDutSik Jan 15, 2025
fc4d4ef
Add now the test for RocksDb.
MathieuDutSik Jan 15, 2025
e25c8e1
Change the structure of the "fn build" and implement the root_key
MathieuDutSik Jan 15, 2025
affe4b2
Rename the admin_test function.
MathieuDutSik Jan 15, 2025
7b2dc86
Make clone_with_root_key async.
MathieuDutSik Jan 15, 2025
5576660
Some insertion of the code.
MathieuDutSik Jan 15, 2025
655d379
Add more tests.
MathieuDutSik Jan 15, 2025
3f5aed2
Formatting corrections and introduce the get_root_keys to the indexddb.
MathieuDutSik Jan 16, 2025
1c93734
Clippy corrections.
MathieuDutSik Jan 16, 2025
72f1ae5
Rename effective_root_key to start_key.
MathieuDutSik Jan 16, 2025
4c82b70
formatting correction.
MathieuDutSik Jan 16, 2025
d40a36f
Several more corrections on the whole code.
MathieuDutSik Jan 16, 2025
c689b9e
Resolve the problem of the rocks_db test.
MathieuDutSik Jan 20, 2025
683ed23
Some simplification and correction of full_keys2 entry.
MathieuDutSik Jan 20, 2025
86742ce
Rename the "get_root_keys" as "list_root_keys".
MathieuDutSik Jan 21, 2025
9c87628
Correct the test code + Correct the DynamoDb/ScyllaDb codes.
MathieuDutSik Jan 21, 2025
fbb0ec7
Some renaming.
MathieuDutSik Jan 22, 2025
29975ab
Cleanup of the code from the review.
MathieuDutSik Jan 22, 2025
3e78982
Add the functionality to the documentation.
MathieuDutSik Jan 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ This document contains the help content for the `linera` command-line program.
* [`linera storage check_absence`↴](#linera-storage-check_absence)
* [`linera storage initialize`↴](#linera-storage-initialize)
* [`linera storage list_namespaces`↴](#linera-storage-list_namespaces)
* [`linera storage list_root_keys`↴](#linera-storage-list_root_keys)

## `linera`

Expand Down Expand Up @@ -937,6 +938,7 @@ Operation on the storage
* `check_absence` — Check absence of a namespace in the database
* `initialize` — Initialize a namespace in the database
* `list_namespaces` — List the namespaces of the database
* `list_root_keys` — List the root keys of the database



Expand Down Expand Up @@ -1012,6 +1014,18 @@ List the namespaces of the database



## `linera storage list_root_keys`

List the root keys of the database

**Usage:** `linera storage list_root_keys --storage <STORAGE_CONFIG>`

###### **Options:**

* `--storage <STORAGE_CONFIG>` — Storage configuration for the blockchain history



<hr/>

<small><i>
Expand Down
36 changes: 18 additions & 18 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ async-lock = "3.3.0"
async-trait = "0.1.77"
async-tungstenite = { version = "0.22", features = ["tokio-runtime"] }
aws-config = "1.1.7"
aws-sdk-dynamodb = "1.16.0"
aws-sdk-dynamodb = "1.60.0"
aws-sdk-s3 = "1.17.0"
aws-smithy-http = "0.60.6"
aws-types = "1.1.7"
Expand Down
9 changes: 9 additions & 0 deletions linera-client/src/client_options.rs
Original file line number Diff line number Diff line change
Expand Up @@ -964,6 +964,14 @@ pub enum DatabaseToolCommand {
#[arg(long = "storage")]
storage_config: String,
},

/// List the root keys of the database
#[command(name = "list_root_keys")]
ListRootKeys {
Comment on lines +969 to +970
Copy link
Contributor

@ma2bd ma2bd Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In linera-client, we know that the "root keys" are (basically) chain ids. So why not list the chain ids instead of abstract root key bytes?

Copy link
Contributor

@ma2bd ma2bd Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And similarly later, we should list ApplicationId, BlobId, etc as such.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw you may need to add conversion function in linera-storage for this.

/// Storage configuration for the blockchain history.
#[arg(long = "storage")]
storage_config: String,
},
}

impl DatabaseToolCommand {
Expand All @@ -975,6 +983,7 @@ impl DatabaseToolCommand {
DatabaseToolCommand::CheckAbsence { storage_config } => storage_config,
DatabaseToolCommand::Initialize { storage_config } => storage_config,
DatabaseToolCommand::ListNamespaces { storage_config } => storage_config,
DatabaseToolCommand::ListRootKeys { storage_config } => storage_config,
};
Ok(storage_config.parse::<StorageConfigNamespace>()?)
}
Expand Down
26 changes: 26 additions & 0 deletions linera-client/src/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,32 @@ impl StoreConfig {
}
}
}

/// Lists all root keys of the storage
pub async fn list_root_keys(self) -> Result<Vec<Vec<u8>>, ViewError> {
match self {
StoreConfig::Memory(_, _) => Err(ViewError::StoreError {
backend: "memory".to_string(),
error: "list_root_keys is not supported for the memory storage".to_string(),
}),
#[cfg(feature = "storage-service")]
StoreConfig::Service(config, namespace) => {
Ok(ServiceStoreClient::list_root_keys(&config, &namespace).await?)
}
#[cfg(feature = "rocksdb")]
StoreConfig::RocksDb(config, namespace) => {
Ok(RocksDbStore::list_root_keys(&config, &namespace).await?)
}
#[cfg(feature = "dynamodb")]
StoreConfig::DynamoDb(config, namespace) => {
Ok(DynamoDbStore::list_root_keys(&config, &namespace).await?)
}
#[cfg(feature = "scylladb")]
StoreConfig::ScyllaDb(config, namespace) => {
Ok(ScyllaDbStore::list_root_keys(&config, &namespace).await?)
}
}
}
}

#[async_trait]
Expand Down
1 change: 1 addition & 0 deletions linera-indexer/lib/src/indexer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ where
let root_key = "indexer".as_bytes().to_vec();
let store = store
.clone_with_root_key(&root_key)
.await
.map_err(|_e| IndexerError::CloneWithRootKeyError)?;
let context = ViewContext::create_root_context(store, ())
.await
Expand Down
1 change: 1 addition & 0 deletions linera-indexer/lib/src/plugin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ where
let root_key = name.as_bytes().to_vec();
let store = store
.clone_with_root_key(&root_key)
.await
.map_err(|_e| IndexerError::CloneWithRootKeyError)?;
let context = ViewContext::create_root_context(store, ())
.await
Expand Down
8 changes: 8 additions & 0 deletions linera-service/src/linera/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1770,6 +1770,14 @@ async fn run(options: &ClientOptions) -> Result<i32, anyhow::Error> {
);
println!("The list of namespaces is {:?}", namespaces);
}
DatabaseToolCommand::ListRootKeys { .. } => {
let root_keys = Box::pin(full_storage_config.list_root_keys()).await?;
info!(
"Root keys listed in {} ms",
start_time.elapsed().as_millis()
);
println!("The list of root keys is {:?}", root_keys);
}
}
Ok(0)
}
Expand Down
37 changes: 19 additions & 18 deletions linera-storage-service/proto/key_value_store.proto
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
syntax = "proto3";
package key_value_store.v1;

import "google/protobuf/empty.proto";

message KeyValue {
bytes key = 1;
bytes value = 2;
Expand Down Expand Up @@ -92,9 +94,6 @@ message RequestWriteBatchExtended {
repeated Statement statements = 1;
}

message ReplyWriteBatchExtended {
}


message RequestSpecificChunk {
int64 message_index = 1;
Expand All @@ -110,9 +109,6 @@ message RequestCreateNamespace {
bytes namespace = 1;
}

message ReplyCreateNamespace {
}


message RequestExistsNamespace {
bytes namespace = 1;
Expand All @@ -127,23 +123,26 @@ message RequestDeleteNamespace {
bytes namespace = 1;
}

message ReplyDeleteNamespace {

message ReplyListAll {
repeated bytes namespaces = 1;
}


message RequestListAll {
message RequestListRootKeys {
bytes namespace = 1;
}

message ReplyListAll {
repeated bytes namespaces = 1;
message ReplyListRootKeys {
repeated bytes root_keys = 1;
}


message RequestDeleteAll {
message RequestInsertRootKey {
bytes namespace = 1;
bytes root_key = 2;
}

message ReplyDeleteAll {
}

service StoreProcessor {
rpc ProcessReadValue (RequestReadValue) returns (ReplyReadValue) {}
Expand All @@ -152,11 +151,13 @@ service StoreProcessor {
rpc ProcessReadMultiValues (RequestReadMultiValues) returns (ReplyReadMultiValues) {}
rpc ProcessFindKeysByPrefix (RequestFindKeysByPrefix) returns (ReplyFindKeysByPrefix) {}
rpc ProcessFindKeyValuesByPrefix (RequestFindKeyValuesByPrefix) returns (ReplyFindKeyValuesByPrefix) {}
rpc ProcessWriteBatchExtended (RequestWriteBatchExtended) returns (ReplyWriteBatchExtended) {}
rpc ProcessWriteBatchExtended (RequestWriteBatchExtended) returns (google.protobuf.Empty) {}
rpc ProcessSpecificChunk (RequestSpecificChunk) returns (ReplySpecificChunk) {}
rpc ProcessCreateNamespace (RequestCreateNamespace) returns (ReplyCreateNamespace) {}
rpc ProcessCreateNamespace (RequestCreateNamespace) returns (google.protobuf.Empty) {}
rpc ProcessExistsNamespace (RequestExistsNamespace) returns (ReplyExistsNamespace) {}
rpc ProcessDeleteNamespace (RequestDeleteNamespace) returns (ReplyDeleteNamespace) {}
rpc ProcessListAll (RequestListAll) returns (ReplyListAll) {}
rpc ProcessDeleteAll (RequestDeleteAll) returns (ReplyDeleteAll) {}
rpc ProcessDeleteNamespace (RequestDeleteNamespace) returns (google.protobuf.Empty) {}
rpc ProcessListAll (google.protobuf.Empty) returns (ReplyListAll) {}
rpc ProcessListRootKeys (RequestListRootKeys) returns (ReplyListRootKeys) {}
rpc ProcessInsertRootKey (RequestInsertRootKey) returns (google.protobuf.Empty) {}
rpc ProcessDeleteAll (google.protobuf.Empty) returns (google.protobuf.Empty) {}
}
Loading
Loading