This week in Databend #20

 Databend[1] aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

•add query log table write by @drmingdrmer[4] (#3468[5])

•add SHOW GRANTS [FOR xxx] statement by @flaneur2020[6]. (#3366[7])

•KVApi support cross nodes read-after-write by @drmingdrmer[8] (#3405[9])

•describe stage by @sundy-li[10]. (#3361[11])

•fuse sys funcs by @dantengsky[12]. (#3302[13])

•[common/meta/sled-store] txn-tree: add update_and_fetch() by @drmingdrmer[14] (#3329[15])

•add strings function

•EXPORT_SET, FIND_IN_SET by @mshauneu[16] (#3343[17] & #3375[18])

•BIN by @dust1[19](#3300[20])

•LENGTH by @BlackSmith96[21](#3448[22])



Improvement

•add expression nullable support by @junli1026[23] (#3452[24])•grant all privileges to the root user of local host by @silathdiir[25] (#3404[26])•support CTAS(Create Table As Select) statement by @junli1026[27] (#3400[28])•github issues/prs table add created/updated/closed time by @Veeupup[29] (#3327[30])•follower forward request to leader •GetDatabase by @drmingdrmer[31] (#3347[32]) •refactoring insert interpreter by @dantengsky[39]. (#3226[40])•ListTables , ListDatabase, GetTableInfo by @ariesdevil[33].(#3360[34], #3369[35])•add more query log events to query log table by @BohuTANG[36].(#3299[37])


Build / Test / CI •refactor the whole ci into stages by @Xuanwo[38] (#3344[39])
•apply the new test style to databend by  @PsiACE[40] (track at #1866[41])•make logs_handler return a bytes stream by @Xuanwo[38](#3344[39])•fix string2string by @mshauneu[44]. (#3332[45])


Bug fixes•make logs_handler return a bytes stream by @Xuanwo[38](#3344[39])•fix string2string by @mshauneu[44]. (#3332[45])

Tips

Let's learn a weekly tip from Databend.

Discuss proposals for Databend

We currently have some discussions about proposals on the GitHub that may help you understand the mechanics of the work or get involved.

•Query Cache

Clever use of caching can provide effective acceleration for Databend. We are implementing a single node memory-disk level 2 cache. Perhaps we will be moving towards distributed caching soon, more discussion on design and implementation is welcome, let's hear your thoughts. Related discussion #3478[46].

•Re-organise our building systems

Our current build / test system is quite complex. We have Makeflie, Dockerfile(s), and a lot of shell / python scripts. We want to build a build/test/benchmark system based on the Rust style, so if you have any good ideas, please feel free to share to us. Related discussion #3419[47].

•Refactor CI pipleline into stages

If you're interested in GitHub workflows, then take a look at this proposal. By refactoring CI[48], we have effectively reduced blocking and been able to maintain it better. Related discussion #3415[49].

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

•v0.6.4-nightly[50]•v0.6.3-nightly[51]•v0.6.2-nightly[52]•v0.6.1-nightly[53]•v0.6.0-nightly[54]•v0.5.41-nightly[55]•v0.5.39-nightly[57]

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

•datafuse-extras/sqlparser-rs #11: add column comment for create table[58] by @Junnplus[59]

•datafuse-extras/sqlparser-rs #12: support negative numbers as value by @sundy-li[61] & @PsiACE[62]

Meet Us

Please join the DatafuseLabs Community[63] if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects. If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues[64] for any problems you find. We also highly appreciate any of your pull requests.

•Databend Website[65]•Twitter[66]•GitHub Discussions[67]•Slack Channel[68]

References

[1] Databend: https://github.com/datafuselabs/databend
[2] Checklist proposal: Nightly v0.6: https://github.com/datafuselabs/databend/issues/2525
[3] Checklist proposal: Nightly v0.7: https://github.com/datafuselabs/databend/issues/3428
[4] @drmingdrmer: https://github.com/drmingdrmer
[5] #3468: https://github.com/datafuselabs/databend/pull/3468
[6] @flaneur2020: https://github.com/flaneur2020
[7] #3366: https://github.com/datafuselabs/databend/pull/3366
[8] @drmingdrmer: https://github.com/drmingdrmer
[9] #3405: https://github.com/datafuselabs/databend/pull/3405
[10] @sundy-li: https://github.com/sundy-li/
[11] #3361: https://github.com/datafuselabs/databend/pull/3361
[12] @dantengsky: https://github.com/dantengsky
[13] #3302: https://github.com/datafuselabs/databend/pull/3302
[14] @drmingdrmer: https://github.com/drmingdrmer
[15] #3329: https://github.com/datafuselabs/databend/pull/3329
[16] @mshauneu: https://github.com/mshauneu
[17] #3343: https://github.com/datafuselabs/databend/pull/3343
[18] #3375: https://github.com/datafuselabs/databend/pull/3375
[19] @dust1: https://github.com/dust1
[20] #3300: https://github.com/datafuselabs/databend/pull/3300
[21] @BlackSmith96: https://github.com/BlackSmith96
[22] #3448: https://github.com/datafuselabs/databend/pull/3448
[23] @junli1026: https://github.com/junli1026
[24] #3452: https://github.com/datafuselabs/databend/pull/3452
[25] @silathdiir: https://github.com/silathdiir
[26] #3404: https://github.com/datafuselabs/databend/pull/3404
[27] @junli1026: https://github.com/junli1026
[28] #3400: https://github.com/datafuselabs/databend/pull/3400
[29] @Veeupup: https://github.com/Veeupup
[30] #3327: https://github.com/datafuselabs/databend/pull/3327
[31] @drmingdrmer: https://github.com/drmingdrmer
[32] #3347: https://github.com/datafuselabs/databend/pull/3347
[33] @ariesdevil: https://github.com/ariesdevil
[34] #3360: https://github.com/datafuselabs/databend/pull/3360
[35] #3369: https://github.com/datafuselabs/databend/pull/3369
[36] @BohuTANG: https://github.com/BohuTANG
[37] #3299: https://github.com/datafuselabs/databend/pull/3299
[38] @Xuanwo: https://github.com/Xuanwo
[39] #3344: https://github.com/datafuselabs/databend/issues/3344
[40] @PsiACE: https://github.com/PsiACE
[41] #1866: https://github.com/datafuselabs/databend/issues/1866
[42] @sunli829: https://github.com/sunli829
[43] #3422: https://github.com/datafuselabs/databend/pull/3422
[44] @mshauneu: https://github.com/mshauneu
[45] #3332: https://github.com/datafuselabs/databend/pull/3332
[46] #3478: https://github.com/datafuselabs/databend/discussions/3478
[47] #3419: https://github.com/datafuselabs/databend/discussions/3419
[48] refactoring CI: https://github.com/datafuselabs/databend/issues/3344
[49] #3415: https://github.com/datafuselabs/databend/discussions/3415
[50] v0.6.4-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.4-nightly
[51] v0.6.3-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.3-nightly
[52] v0.6.2-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.2-nightly
[53] v0.6.1-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.1-nightly
[54] v0.6.0-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.6.0-nightly
[55] v0.5.41-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.5.41-nightly
[56] v0.5.40-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.5.40-nightly
[57] v0.5.39-nightly: https://github.com/datafuselabs/databend/releases/tag/v0.5.39-nightly
[58] datafuse-extras/sqlparser-rs #11: add column comment for create table: https://github.com/datafuse-extras/sqlparser-rs/pull/11
[59] @Junnplus: https://github.com/Junnplus/
[60] datafuse-extras/sqlparser-rs #12: support negative numbers as value: https://github.com/datafuse-extras/sqlparser-rs/pull/12
[61] @sundy-li: https://github.com/sundy-li/
[62] @PsiACE: https://github.com/PsiACE/
[63] DatafuseLabs Community: https://github.com/datafuselabs/
[64] issues: https://github.com/datafuselabs/databend/issues
[65] Databend Website: https://databend.rs
[66] Twitter: https://twitter.com/Datafuse_Labs
[67] GitHub Discussions: https://github.com/datafuselabs/databend/discussions
[68] Slack Channel: https://datafusecloud.slack.com/join/shared_invite/zt-nojrc9up-50IRla1Y1h56rqwCTkkDJA

Comments

Post a Comment

Popular posts from this blog

The Databend Community Actively Embraces The Rust Ecosystem

Image
The Databend Community Actively Embraces The Rust Ecosystem Databend aims to become an open source, flexible, and reliable serverless data warehouse, it can query fast as lightning, and organically combined with flexible, simple, and low-cost cloud services. The construction of data cloud has never been easier! What Databend Means to The Rust Community Most Databend's team members have rich experience in database/data platform development and open source collaboration. So we choose Rust to develop the next-generation data cloud services which can performance higher performance and more reliable. 1.  Create the top seed player in the Rust data warehouse field, show the charm of Rust, and promote the further application of Rust in the database field. 2.  Continue to promote Rust's efforts in data governance/database/cloud native and other domains to improve the ecosystem and tools. 3.  Provide more learning/practice/job opportunities for Rustacean, and cultivate a group of ...
Read more

This week in Databend #26

[Databend](https://github.com/datafuselabs/databend) aimed to be an open source **elastic** and **reliable** cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy. ## Big changes Below is a list of some major changes that we don't want you to miss. ### Features - **add bloom filter for single column equivalent expression** by [@junli1026](https://github.com/junli1026) ([#3887](https://github.com/datafuselabs/databend/pull/3887)) - **Jwt Auth for http handler** by [@youngsofun](https://github.com/youngsofun) ([#3993](https://github.com/datafuselabs/databend/pull/3993)) ## Improvement - **add partitions_scanned and partitions_total to planner statistics** by [@BohuTANG](https://github.com/BohuTANG). ([#3909](https://github.com/datafuselabs/databend/pull/3909)) - **clean up metasrv code and test** by [@drmingdrmer](https://github.com/drmingdrmer). ([#3925](https://github.com/datafuselabs/databend/pul...
Read more

Databend Design | Whitepapers

Image
Databend is an open source   elastic   and   scalable   cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy. Databend is intended for executing workloads with data stored in cloud storage systems, such as AWS S3 and Azure Blob Storage or others. We design Databend with the following key functionalities in mind: Elastic  In Databend storage and compute resources can dynamically scale up and down on demand. Secure  All data files and network traffic in Databend is encrypted end-to-end, and provider  Role Based Access Control  in SQL level. User-friendly  Databend is an ANSI SQL compliant cloud warehouse, it is easy for data scientist and engineers to use. Cost-efficient  Databend processes queries with high performance, and the user only pays for what is actually used. The picture above shows the high-level architecture of Databend, it consists of three...
Read more