.NET for Apache Spark provides high performance APIs for using Apache Spark from C# and F#. With these .NET APIs, you can access the most popular Dataframe and SparkSQL aspects of Apache Spark, for working with structured data, and Spark Structured Streaming, for working with streaming data.
.NET for Apache Spark is compliant with .NET Standard - a formal specification of .NET APIs that are common across .NET implementations. This means you can use .NET for Apache Spark anywhere you write .NET code allowing you to reuse all the knowledge, skills, code, and libraries you already have as a .NET developer.
.NET for Apache Spark runs on Windows, Linux, and macOS using .NET 8, or Windows using .NET Framework. It also runs on all major cloud providers including Azure HDInsight Spark, Amazon EMR Spark, AWS & Azure Databricks.
Note: We currently have a Spark Project Improvement Proposal JIRA at SPIP: .NET bindings for Apache Spark to work with the community towards getting .NET support by default into Apache Spark. We highly encourage you to participate in the discussion.
Apache Spark | .NET for Apache Spark |
---|---|
2.4* | v2.1.1 |
3.0 | |
3.1 | |
3.2 | |
3.5 |
*2.4.2 is not supported.
.NET for Apache Spark releases are available here and NuGet packages are available here.
These instructions will show you how to run a .NET for Apache Spark app using .NET 8.
Ubuntu | Windows |
Building from source is very easy and the whole process (from cloning to being able to run your app) should take less than 15 minutes!
Instructions | ||
---|---|---|
Windows |
|
|
Ubuntu |
There are two types of samples/apps in the .NET for Apache Spark repo:
Getting Started - .NET for Apache Spark code focused on simple and minimalistic scenarios.
End-End apps/scenarios - Real world examples of industry standard benchmarks, usecases and business applications implemented using .NET for Apache Spark.
We welcome contributions to both categories!
Analytics Scenario |
Description |
Scenarios |
Dataframes and SparkSQL |
Simple code snippets to help you get familiarized with the programmability experience of .NET for Apache Spark. |
Basic C# F# |
Structured Streaming |
Code snippets to show you how to utilize Apache Spark's Structured Streaming (2.3.1, 2.3.2, 2.4.1, Latest) |
Word Count C# F#Windowed Word Count C# F#Word Count on data from Kafka C# F# |
TPC-H Queries |
Code to show you how to author complex queries using .NET for Apache Spark. |
TPC-H Functional C#TPC-H SparkSQL C# |
We welcome contributions! Please review our contribution guide.
This project would not have been possible without the outstanding work from the following communities:
The .NET for Apache Spark team encourages contributions, both issues and PRs. The first step is finding an existing issue you want to contribute to or if you cannot find any, open an issue.
.NET for Apache Spark is an open source project under the .NET Foundation and does not come with Microsoft Support unless otherwise noted by the specific product. For issues with or questions about .NET for Apache Spark, please create an issue. The community is active and is monitoring submissions.
The .NET for Apache Spark project is part of the .NET Foundation.
This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.
.NET for Apache Spark is licensed under the MIT license.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。