克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

DCluster

该项目是基于dolphinscheduler海豚调度之上的数据开发治理服务分析一站式数据中台。平台旨在研发数据集成、数据工作流离线开发、实时开发、数据资产治理、指标管理、智能分析Copilot、数据质量、数据分析等核心功能。正在以及规划集成优秀的开源项目dinky、supersonic、datavines、数据血缘等。

目录

[TOC]

智能ChatBi&Copilot(集成supersonic)

  • 多指标联合问答:基于指标模型支持单个指标问答以及多个指标多个维度联合问答
  • 智能补全:支持输入搜索词后返回联想指标和维度,帮助快速补全
  • 历史会话:对历史记录进行管理

指标管理

  • 指标模型管理:快速灵活组合各个指标和维度,提供API和智能问答的指标语义模型
  • 指标定义管理:支持原子指标和衍生指标的定义,统一指标口径指标
  • 维度管理:对业务维度进行管理,标准规范化维度管理

数据质量管理(Datavines)

  • 数据质量监控:定义和执行数据质量规则,如空值检测、重复值检测、范围检查等,确保数据的准确性和完整性。
  • 告警和通知:当数据质量出现问题时,Datavines通过电子邮件、短信等方式发送告警通知,确保相关人员能够及时响应和处理问题。
  • 数据治理:支持数据分类、标签和权限管理,确保数据的合规性和安全性。
  • 数据质量报告:生成详细的数据质量报告,帮助用户了解和改进数据质量。

工作流调度(DolphinScheduler)

  • 分布式架构:处理大规模的数据处理任务,确保系统的高可用性和扩展性。
  • 任务依赖管理:定义复杂的任务依赖关系,确保任务按照预定的顺序执行。 数据集成与开发:支持多种类型的任务,包括Shell、Python 脚本、SQL、MapReduce、Spark、Flink 等。
  • 可视化工作流设计:提供直观的图形化用户界面,通过拖拽方式设计和配置工作流。
  • 任务调度:支持灵活的调度策略,包括定时调度、依赖触发、手动触发等。 任务监控与告警:实时监控任务的执行状态,并提供告警通知功能,确保问题及时处理。
  • 资源管理:管理任务所需的资源,确保资源合理分配和使用。

实时数据处理(Dinky)

  • 实时数据处理:Dinky 提供了强大的实时数据处理能力,支持Flink等流处理框架。
  • 实时计算任务:用户可以定义和执行实时计算任务,处理实时数据流。 数据集成:支持多种数据源和目标,能够将实时数据流集成到不同的数据系统中。 可视化开发:提供直观的图形化界面,用户可以通过拖拽方式设计和配置实时计算任务。
  • 监控与告警:实时监控数据流处理状态,并在出现异常时提供告警通知。 高可用性:设计了高可用性和容错机制,确保实时数据处理的稳定性和可靠性。

数据集成

  • 多数据源支持:支持多种数据源的集成,包括数据库、API、文件系统等,能够灵活应对不同的数据来源。
  • 集成任务依赖支持:支持下游开发任务依赖集成同步任务,支持数据导出外部数据源任务依赖数据开发任务。
  • 多执行引擎支持:支持多种执行引擎,包括DataX、seatunnal、flinkx等,满足不同场景的数据同步需求。
  • 全库同步:支持整个数据库的同步,方便全面的数据迁移和复制。
  • .多表同步:支持多表同步,简化复杂的数据集成任务。 全量和增量同步:提供全量和增量同步功能,实现数据的一体化同步。
  • 断点续传:支持断点续传机制,确保数据集成过程的可靠性和稳定性。

数据查询

  • 灵活高效的数据查询:支持多种数据源进行查询。 多数据源支持:支持多种数据源的数据查询,适用于不同的数据存储系统。
  • 高性能:优化的查询引擎,确保数据查询的高性能和低延迟。

数据分析

  • 集成多种数据分析工具:集成多种数据分析和可视化工具,帮助用户快速获取洞见。 可视化分析:提供丰富的可视化组件,用户可以通过图表、仪表板等方式展示分析结果。
  • 高级分析功能:支持机器学习模型、统计分析等高级分析功能,满足复杂数据分析需求。

数据服务

  • 数据服务接口:提供数据服务接口,支持RESTful API,方便数据的对外提供和集成。
  • 灵活的数据访问:通过API提供灵活的数据访问方式,满足不同应用和系统的集成需求。
  • 高可用性和安全性:设计了高可用性和安全性机制,确保数据服务的可靠性和数据的安全性。

安装&演示

演示环境:http://36.155.14.171:12345/dolphinscheduler/ui/ 账号:admin
密码:dcluster123 在线文档:http://47.121.127.33:8090/

功能截图

输入图片说明

输入图片说明

输入图片说明

输入图片说明

输入图片说明

输入图片说明

输入图片说明

输入图片说明

输入图片说明

社区联系方式

管理员微信:Lq-time 输入图片说明

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ======================================================================= Apache DolphinScheduler Subcomponents: The Apache DolphinScheduler project contains subcomponents with separate copyright notices and license terms. Your use of the source code for the these subcomponents is subject to the terms and conditions of the following licenses. ======================================================================== Apache 2.0 licenses ======================================================================== The following components are provided under the Apache License. See project link for details. The text of each license is the standard Apache 2.0 license. ScriptRunner from https://github.com/mybatis/mybatis-3 Apache 2.0 mvnw files from https://github.com/apache/maven-wrapper Apache 2.0 PropertyPlaceholderHelper from https://github.com/spring-projects/spring-framework Apache 2.0 DolphinPluginClassLoader from https://github.com/prestosql/presto Apache 2.0 DolphinPluginDiscovery from https://github.com/prestosql/presto Apache 2.0 DolphinPluginLoader from https://github.com/prestosql/presto Apache 2.0 CodeGenerateUtils from https://github.com/twitter-archive/snowflake/tree/snowflake-2010 Apache 2.0

简介

dcluster是一款致力于开发从数据集成到数据开发治理再到数据智能分析的一站式服务平台。通过集成优秀的开源项目dolphinscheduler、supersonic、dinky等实现任务调度、实时开发、智能分析等核心功能。 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

不能加载更多了
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化