The Databend Community Actively Embraces The Rust Ecosystem

The Databend Community Actively Embraces The Rust Ecosystem

Databend aims to become an open source, flexible, and reliable serverless data warehouse, it can query fast as lightning, and organically combined with flexible, simple, and low-cost cloud services. The construction of data cloud has never been easier!

What Databend Means to The Rust Community

Most Databend's team members have rich experience in database/data platform development and open source collaboration. So we choose Rust to develop the next-generation data cloud services which can performance higher performance and more reliable.

1. Create the top seed player in the Rust data warehouse field, show the charm of Rust, and promote the further application of Rust in the database field.

2. Continue to promote Rust's efforts in data governance/database/cloud native and other domains to improve the ecosystem and tools.

3. Provide more learning/practice/job opportunities for Rustacean, and cultivate a group of database development experts for the community.

Close Connection Between Databend and The Rust Ecosystem

Since the first day to establish the Databend community, we have embraced the belief of "from the community, to the community" and actively embracing open source and the Rust ecosystem.

Course

Since August, Databend, together with the Rust Chinese community and Zhishutang, had launched a public course plan for Rustacean and database developers.

 

Rust Training and Improvement Plan

• Know the basic architecture language Rust: https://www.bilibili.com/video/BV1mg411778g

• Understand the Rust's ownership: https://www.bilibili.com/video/BV1264y1i7U9

• Practice to understand the macros in Rust | 90 minutes to give you a deepen understanding of macros in Rust：https://www.bilibili.com/video/BV1Yb4y1U7r1

• Understanding Omni-Knowledge Tracking through Datafuse: https://www.bilibili.com/video/BV1YA411c7ia

• Introduction to Rust Asynchronous Programming-Futures：https://www.bilibili.com/video/BV1mf4y1N7MJ

• Introduction to Tokio at runtime: https://www.bilibili.com/video/BV1oy4y1G7jC

• Deepen understanding of Rust closures: https://www.bilibili.com/video/BV1Zf4y1J7jq

• Use Tokio to develop a high-performance Mini Http Server: https://www.bilibili.com/video/BV143411q79B

• Develop short-chain services based on Axum-web and MySQL: https://www.bilibili.com/video/BV143411q79B

• Use Redis to accelerate short-chain services: https://www.bilibili.com/video/BV1iQ4y1B7qX

• Use Rust to develop a dynamic link library and applied to Golang：https://www.bilibili.com/video/BV1WT4y1o72d

• Explore the Rust microservice architecture: https://www.bilibili.com/video/BV1jL4y1B7rK

• Explore the Rust asynchronous programming framework Mio：https://www.bilibili.com/video/BV1P3411k7tL

 Databend column 

• The new generation cloud native data warehouse Databend architecture and status: https://www.bilibili.com/video/BV1dh411b7Vv

• How to test and control Databend products：https://www.bilibili.com/video/BV1CL411G7TW 

Upstream

The success of Databend can’t separate off the prosperity Rust ecosystem and high-quality upstream projects, the Databend community will also provide patches to upstream from time to time, to build a better Rust ecosystem.

• Advisory-db – is a security advisory database created for Rust crate witch published in crates.io.

• RUSTSEC-2020-0009 added flatbuffers patch.

• Arrow-rs & arrow-flight & parquet - Apache Arrow / Parquet which official Rust implementation, as Databend early important dependency. 

￮ For sort added limit options.

￮ For the affect of RUSTSEC-2020-0009, updated flatbuffers to the latest version.

￮ Prettytable-rs was in an unmaintained state, replaced by comfy-table.

￮ Updated prost to 0.8, and updated the corresponding tonic.

￮ Updated prost to 0.9, and updated the corresponding tonic, debuged error.

• Arrow2 & arrow-format & parquet2 - use transmute-free operational rewritten Arrow and the fastest and safest (unsafe free) Parquet implementation.

￮ For merge sort added limit options.

￮ Improved sort.

￮ Added wrapping_cast to improve cast performance.

￮ Added BinaryArray computing support.

￮ prettytable-rs was in an unmaintained state, replaced by comfy-table.

￮ Update prost to 0.8, and updated the corresponding tonic.

￮ Update prost to 0.9, and updated the corresponding tonic, debuged error clippy lints.

• Async-raft - based on tokio to implement asynchronous raft consensus algorithm.

￮ Debug: NonVoter should keep its identity instead of turning to Follower when restart.

￮ A series of additional repairs had been list in https://github.com/async-raft/async-raft/issues/132#issuecomment-915853535.

￮ Due to the async-raft native library is not well maintained, especially datafuse-extras/async-raft.

• Cargo-license - Utility tool for viewing dependencies on adopted license agreements.

￮ Update dependencies to avoid conflicts, repaired clippy lints.

• Clickhouse-srv - used to write a binding, it can act as a ClickHouse server.

￮ This project had been take over by Databend team to develop and protect.

￮ Had been merged into Databend project, in common/clickhouse-srv.

• Foat-cmp - approximate comparison of floating-point numbers.

￮ Specify the minimum version of num-traits, repaired clippy lints and to help upstream dependencies pass minver test.

• Hashbrown – high performance HashMap, it’s also a part of standard library.

￮ Updated rand to 0.8.3

• lru-rs - Lru Cache implement. 

￮ Updateed hashbrown to 0.11.2, repaired clippy lints.

• Mockall - for easy writing the tests practical library.

￮ Update the dependencies which affected by RUSTSEC-2020-0095.

• Msql-srv - Used to write a binding ,it can act as a MySQL/MariaDB server. 

￮ Repaired the corresponding of COM_FIELD_LIST.

￮ Synchronized the dependency to the latest version, and repaired the dependency in the test.

￮ Added basic AuthPlugin support.

￮ Query progress information statistics.

￮ Part of the patch is trying to be merged upstream, you can try it in datafuse-extras/msql-srv.

• Prof - CPU / Application performance testing tool.

￮ Updated prost to 0.8, and updated the corresponding tonic.

￮ Updated prost to 0.9, and updated the corresponding tonic, repaird clippy lints.

• Ritelinked - quickly & useful LinkedHashMap & LinkedHashSet implement. 

￮ This project had been take over by Databend team to develop and protect.

￮ Currently at ritelabs/ritelinked.

• Sqlparser-rs - extensible SQL lexical analyzer and syntax parser.

￮ Use return error to replace panic.

￮ Added the Fuzz Testing tools which based on honggfuzz.

￮ Support tinyint.

￮ Datafuse-extras/sqlparser-rs includes Databend customized features. 

▪ Support  INSERT FORMAT.

▪ Support cast(a, b).

▪ Make the function support parameters.

▪ Support substring(string, start, length) grammar.

• Structopt-toml – structopt’s TOML default value loading tool. 

￮ Failure was in an unmaintained state, replaced by anyhow.

Databend's community development plan

The Databend community is a platform which built by every follower, developer, and user. In the future, we will continue to improve the feedback mechanism to help each community member grow and progress in the process of community building.

1. Establish and improve the developer growth system to provide Rustacean with a platform which can learning, practice and innovation together.

2. Continue to output high-quality Rust + database courses, invite industry experts to share experience, and promote high-quality projects in the Databend ecosystem.

3. Provide sponsorship, training and other opportunities for the important upstream and downstream projects and high-quality developers to encourage their contribution in open source and ecosystem.

The name of the Databend is d from the theory of relativity. Due to the existence of matter, time and space will bend, which is the Time Bend of relativity. Because of the Databend we hope users can re-examine the data, and mining the greater value in the data, at the same time, let users in the field of big data: Make life easier.

Welcome to join the Databend community and work with us to build the next generation of data cloud services with Rust.