克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

StreamX logo

Make Flink|Spark easier!!!

##English | 中文

StreamX

Make Flink|Spark easier!!!

一个神奇的框架,让 Flink|Spark 开发更简单

🚀 什么是StreamX

    大数据技术如今发展的如火如荼,已经呈现百花齐放欣欣向荣的景象,实时处理流域 Apache SparkApache Flink 更是一个伟大的进步,尤其是 Apache Flink 被普遍认为是下一代大数据流计算引擎, 我们在使用 Flink 时发现从编程模型, 启动配置到运维管理都有很多可以抽象共用的地方, 我们将一些好的经验固化下来并结合业内的最佳实践, 通过不断努力终于诞生了今天的框架 —— StreamX, 项目的初衷是 —— 让 Flink 开发更简单, 使用 StreamX 开发,可以极大降低学习成本和开发门槛, 让开发者只用关心最核心的业务, StreamX 规范了项目的配置,鼓励函数式编程,定义了最佳的编程方式,提供了一系列开箱即用的 Connectors ,标准化了配置、开发、测试、部署、监控、运维的整个过程, 提供 ScalaJava 两套api, 其最终目的是打造一个一站式大数据平台,流批一体,湖仓一体的解决方案

StreamX video

🎉 Features

  • 开发脚手架
  • 一系列开箱即用的connectors
  • 支持项目编译功能(maven 编译)
  • 在线参数配置
  • 多版本flink支持(1.12.x,1.13.x,1.14.x)
  • on Kubernetes部署模式支持(K8s-Native-Application/K8s-Native-Session)
  • on YARN部署模式支持(YARN-Application/YARN-Per-Job)
  • 快捷的日常操作(任务启动停止savepoint,从savepoint恢复)
  • 支持火焰图
  • 支持 notebook (在线任务开发)
  • 项目配置和依赖版本化管理
  • 支持任务备份、回滚(配置回滚)
  • 在线管理依赖(maven pom)和自定义jar
  • 自定义 udf、连接器等支持
  • Flink SQL WebIDE
  • 支持 Catalog、Hive
  • 从任务开发阶段到部署管理全链路支持
  • ...

🏳‍🌈 组成部分

Streamx 由三部分组成,分别是 streamx-corestreamx-pumpstreamx-console

1️⃣ streamx-core

streamx-core 定位是一个开发时框架,关注编码开发,规范了配置文件,按照约定优于配置的方式进行开发,提供了一个开发时 RunTime Content 和一系列开箱即用的 Connector ,扩展了 DataStream 相关的方法,融合了DataStreamFlink sql api,简化繁琐的操作,聚焦业务本身,提高开发效率和开发体验

2️⃣ streamx-pump

pump 是抽水机,水泵的意思,streamx-pump 的定位是一个数据抽取的组件,类似于 flinkx,基于streamx-core 中提供的各种 connector 开发,目的是打造一个方便快捷,开箱即用的大数据实时数据抽取和迁移组件,并且集成到 streamx-console 中,解决实时数据源获取问题,目前在规划中

3️⃣ streamx-console

streamx-console 是一个综合实时数据平台,低代码(Low Code)平台,可以较好的管理Flink任务,集成了项目编译、发布、参数配置、启动、savepoint,火焰图(flame graph) ,Flink SQL, 监控等诸多功能于一体,大大简化了 Flink 任务的日常操作和维护,融合了诸多最佳实践。旧时王谢堂前燕,飞入寻常百姓家,让大公司有能力研发使用的项目,现在人人可以使用, 其最终目标是打造成一个实时数仓,流批一体的一站式大数据解决方案,该平台使用但不仅限以下技术:

感谢以上优秀的开源项目和很多未提到的优秀开源项目,致以最崇高的敬意,特别感谢Apache ZeppelinIntelliJ IDEA, 感谢 fire-spark 项目,早期给予的灵感和帮助, 感谢我老婆在项目开发时给予的支持,悉心照顾我的生活和日常,给予我足够的时间开发这个项目

🚀 快速上手

git clone https://github.com/streamxhub/streamx.git
cd streamx
mvn clean install -DskipTests -Denv=prod

更多请查看官网文档

👻 为什么不是...❓

Apache Zeppelin

Apache Zeppelin是一个非常优秀的开源项目👏 对 Flink 做了很好的支持,Zeppelin 创新型的 notebook 功能,让开发者非常方便的 On-line 编程,快捷的提交任务,语言层面同时支持JavaScalaPython,国内阿里的章剑峰大佬也在积极推动该项目,向剑峰大佬致以崇高的敬意🙏🙏🙏, 但该项目目前貌似没有解决项目的管理和运维方面的痛点,针对比较复杂的项目和大量的作业管理就有些力不从心了,一般来讲不论是 DataStream 作业还是 Flink SQL 作业,大概都会经历作业的开发阶段测试阶段打包阶段上传服务器阶段启动任务阶段等这些步骤,这是一个链路很长的步骤,且整个过程耗时比较长,体验不好, 即使修改了一个符号,项目改完上线都得走上面的流程,我们期望这些步骤能够动动鼠标一键式解决,还希望至少能有一个任务列表的功能,能够方便的管理任务,可以清楚的看到哪些任务正在运行,哪些停止了,任务的资源消耗情况,可以在任务列表页面一键启动停止任务,并且自动管理 savePoint,这些问题也是开发者实际开发中会遇到了问题, streamx-console 很好的解决了这些痛点,定位是一个一站式实时数据平台,并且开发了更多令人激动的功能(诸如Flink SQL WebIDE依赖隔离任务回滚火焰图等)

FlinkX

FlinkX 是基于flink的分布式数据同步工具,实现了多种异构数据源之间高效的数据迁移,定位比较明确,专门用来做数据抽取和迁移,可以作为一个服务组件来使用,StreamX 关注开发阶段和任务后期的管理,定位有所不同,streamx-pump 模块也在规划中, 致力于解决数据源抽取和迁移,最终会集成到 streamx-console

🤝 如何贡献

PRs Welcome

如果你希望参与贡献 欢迎 Pull Request,或给我们 报告 Bug

强烈推荐阅读 《提问的智慧》(本指南不提供此项目的实际支持服务!)、《如何有效地报告 Bug》《如何向开源项目提交无法解答的问题》,更好的问题更容易获得帮助。

感谢 JetBrains 提供的免费开源 License 赞助

JetBrains

💰 Donation

Are you enjoying this project ? 👋

StreamX 采用 Apache LICENSE 2.0 许可的开源项目,使用完全免费, 旧时王谢堂前燕,飞入寻常百姓家,让大公司有能力研发使用的项目,现在人人都可以使用,流批一体,湖仓一体,数据湖是大数据领域的趋势, StreamX现在离这个目标还有一段距离,还需持续投入,相应的资金支持能更好的持续项目的维护和开发。你可以通过下列的方法来赞助 StreamX 的开发, 欢迎捐助,一起来帮我们做的更好! ☀️ 👊

微信支付 支付宝
Buy Me A Coffee Buy Me A Coffee

所有赞助人将在赞助人名单中显示。

🏆 我的赞助

💜 月度赞助商

虚席以待,欢迎个人和企业前来赞助,您的支持会帮助我们更好的发展项目

🥇 金牌赞助商

🥈 白金赞助商

🥈 白银赞助商

💬 加入社区

Streamx 已正式开源,现已经进入高速发展模式,如果您觉得还不错请在右上角点一下 star,帮忙转发,谢谢 🙏🙏🙏 大家的支持是开源最大动力, 你可以扫下面的二维码加入官方微信群,更多相关信息请访问官网

Stargazers over time

关注我们
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2021 http://www.streamxhub.com Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

StreamX 是一个 Apache Flink 极速开发框架 展开 收起
Java
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

不能加载更多了
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化