This week in Databend #23

Databend[1] aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

•add proxy mode access check by @BohuTANG[2]. (#3776[3])•add basic tokio-console support by @PsiACE[4]. (#3739[5])•add generate command for bendctl by @kevinw66[6]. (#3732[7])

Improvement

•add more metrics to query_log by @dantengsky[8]. (#3773[9])•make parquet reader works with only one stream reader by @dantengsky[10]. (#3757[11])•fuse metadata cache by @Veeupup[12]. (#3761[13])•UDF refinement by @lianghanzhen[14]. (#3743[15])•cli up improvement by @sundy-li[16]. (#3699[17])•replace arrow flight with gRPC by @ariesdevil[18]. (#3606[19])•improve origin array builder & make nullable/no-nullable builder support by @Veeupup[20]. (#3700[21])

Performance Improvement

•improve performance of arithmetic functions by @zhyass[22] (#3697[23])•table read performance tweaking by @dantengsky[24]. (#3718[25])

Bug fixes

•support as case-insensitive when create udf by @kevinw66[26]. (#3703[27])

Tips

Let's learn a weekly tip from Databend.

Tracing in Databend

Databend using Rust's tracing ecosystem tokio-tracing[28] to do log and profile.

Distributed tracing with Jaeger

Jaeger[29], is a distributed tracing platform. It can be used for monitoring microservices-based distributed systems.

•4 steps to enable Jaeger monitor•build databend-query: cargo build --bin databend-query•run with DEBUG log level: LOG_LEVEL=DEBUG ./databend-query•start jaeger: docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest•Open http://127.0.0.1:16686/•Jaeger Tracing Show

jaeger-tracing-show
jaeger-tracing-show

•Read More: https://databend.rs/dev/development/tracing#distributed-tracing-with-jaeger

Explore and diagnose with tokio-console

tokio-console[30] is a diagnostics and debugging tool for asynchronous Rust programs.

•3 steps to enable console subscriber•build databend-query with rustflags & features: RUSTFLAGS="--cfg tokio_unstable" cargo build --bin databend-query --features tokio-console•run with the log level of TRACE: LOG_LEVEL=TRACE databend-query•run tokio-console•Run tokio-console to explore databend-query

query-console
query-console

•Read More: https://databend.rs/dev/development/tracing#explore-and-diagnose-with-tokio-console

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

•v0.6.25-nightly[31]•v0.6.24-nightly[32]•v0.6.23-nightly[33]•v0.6.22-nightly[34]•v0.6.21-nightly[35]•v0.6.20-nightly[36]•v0.6.19-nightly[37]

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

•arrow2 #73: add cargo check to benchmarks[38] by @sundy-li[39].

Meet Us

Please join the DatafuseLabs Community[40] if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects.

If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues[41] for any problems you find. We also highly appreciate any of your pull requests.

•Databend Website[42]•Twitter[43]•GitHub Discussions[44]•Slack Channel[45]

引用链接

[1] Databend: https://github.com/datafuselabs/databend
[2] @BohuTANG: https://github.com/BohuTANG
[3] #3776: https://github.com/datafuselabs/databend/pull/3776
[4] @PsiACE: https://github.com/PsiACE
[5] #3739: https://github.com/datafuselabs/databend/pull/3739
[6] @kevinw66: https://github.com/kevinw66
[7] #3732: https://github.com/datafuselabs/databend/pull/3732
[8] @dantengsky: https://github.com/dantengsky
[9] #3773: https://github.com/datafuselabs/databend/pull/3773
[10] @dantengsky: https://github.com/dantengsky
[11] #3757: https://github.com/datafuselabs/databend/pull/3757
[12] @Veeupup: https://github.com/Veeupup
[13] #3761: https://github.com/datafuselabs/databend/pull/3761
[14] @lianghanzhen: https://github.com/lianghanzhen
[15] #3743: https://github.com/datafuselabs/databend/pull/3743
[16] @sundy-li: https://github.com/sundy-li/
[17] #3699: https://github.com/datafuselabs/databend/pull/3699
[18] @ariesdevil: https://github.com/ariesdevil/
[19] #3606: https://github.com/datafuselabs/databend/pull/3606
[20] @Veeupup: https://github.com/Veeupup
[21] #3700: https://github.com/datafuselabs/databend/pull/3700
[22] @zhyass: https://github.com/zhyass
[23] #3697: https://github.com/datafuselabs/databend/pull/3697
[24] @dantengsky: https://github.com/dantengsky
[25] #3718: https://github.com/datafuselabs/databend/pull/3718
[26] @kevinw66: https://github.com/kevinw66
[27] #3703: https://github.com/datafuselabs/databend/pull/3703
[28] tokio-tracing: https://github.com/tokio-rs/tracing
[29] Jaeger: https://github.com/jaegertracing/jaeger
[30] tokio-console: https://github.com/tokio-rs/console
[31] v0.6.25-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.25-nightly
[32] v0.6.24-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.24-nightly
[33] v0.6.23-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.23-nightly
[34] v0.6.22-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.22-nightly
[35] v0.6.21-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.21-nightly
[36] v0.6.20-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.20-nightly
[37] v0.6.19-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.19-nightly
[38] arrow2 #73: add cargo check to benchmarks: https://github.com/jorgecarleitao/arrow2/pull/730
[39] @sundy-li: https://github.com/sundy-li/
[40] DatafuseLabs Community: https://github.com/datafuselabs/
[41] issues: https://github.com/datafuselabs/databend/issues
[42] Databend Website: https://databend.rs
[43] Twitter: https://twitter.com/Datafuse_Labs
[44] GitHub Discussions: https://github.com/datafuselabs/databend/discussions
[45] Slack Channel: https://datafusecloud.slack.com/join/shared_invite/zt-nojrc9up-50IRla1Y1h56rqwCTkkDJA

Comments

Post a Comment

Popular posts from this blog

The Databend Community Actively Embraces The Rust Ecosystem

Image
The Databend Community Actively Embraces The Rust Ecosystem Databend aims to become an open source, flexible, and reliable serverless data warehouse, it can query fast as lightning, and organically combined with flexible, simple, and low-cost cloud services. The construction of data cloud has never been easier! What Databend Means to The Rust Community Most Databend's team members have rich experience in database/data platform development and open source collaboration. So we choose Rust to develop the next-generation data cloud services which can performance higher performance and more reliable. 1.  Create the top seed player in the Rust data warehouse field, show the charm of Rust, and promote the further application of Rust in the database field. 2.  Continue to promote Rust's efforts in data governance/database/cloud native and other domains to improve the ecosystem and tools. 3.  Provide more learning/practice/job opportunities for Rustacean, and cultivate a group of ...
Read more

This week in Databend #26

[Databend](https://github.com/datafuselabs/databend) aimed to be an open source **elastic** and **reliable** cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy. ## Big changes Below is a list of some major changes that we don't want you to miss. ### Features - **add bloom filter for single column equivalent expression** by [@junli1026](https://github.com/junli1026) ([#3887](https://github.com/datafuselabs/databend/pull/3887)) - **Jwt Auth for http handler** by [@youngsofun](https://github.com/youngsofun) ([#3993](https://github.com/datafuselabs/databend/pull/3993)) ## Improvement - **add partitions_scanned and partitions_total to planner statistics** by [@BohuTANG](https://github.com/BohuTANG). ([#3909](https://github.com/datafuselabs/databend/pull/3909)) - **clean up metasrv code and test** by [@drmingdrmer](https://github.com/drmingdrmer). ([#3925](https://github.com/datafuselabs/databend/pul...
Read more

Databend Design | Whitepapers

Image
Databend is an open source   elastic   and   scalable   cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy. Databend is intended for executing workloads with data stored in cloud storage systems, such as AWS S3 and Azure Blob Storage or others. We design Databend with the following key functionalities in mind: Elastic  In Databend storage and compute resources can dynamically scale up and down on demand. Secure  All data files and network traffic in Databend is encrypted end-to-end, and provider  Role Based Access Control  in SQL level. User-friendly  Databend is an ANSI SQL compliant cloud warehouse, it is easy for data scientist and engineers to use. Cost-efficient  Databend processes queries with high performance, and the user only pays for what is actually used. The picture above shows the high-level architecture of Databend, it consists of three...
Read more