apache/spark ? reverse-engineered prompt

Reverse engineered prompt

Build me a unified analytics engine for large-scale data processing. The main idea is to create a powerful framework that developers can use with multiple languages like Python, Java, Scala, and R. It should be optimized for handling huge datasets. The engine needs to include several key components: a library for running SQL queries (like Spark SQL), a machine learning library (MLlib), a module for graph processing (GraphX), and support for real-time structured streaming. It's also important that it has a familiar API for Python users, maybe something compatible with pandas. Basically, I want a complete system for all kinds of big data analysis, from simple queries to complex machine learning pipelines.

Are you gonna build this?

make sure you review the code using coderabbit

Try freeSponsored — opens CodeRabbit in a new tab