DaniyalAneeq/stockmarket-kafka-project — reverse-engineered prompt

Build me a working demo project for a real time stock market data pipeline. I want to stream rows from a stock market CSV as if they are live data, send them into Apache Kafka running on an AWS EC2 instance, then have a consumer read the messages and save them into an S3 bucket so I can catalog them with Glue and query them in Athena.

Please make it beginner friendly. Include clear Jupyter notebooks for the producer and consumer, a requirements file, a sample env file, and setup docs that walk me through creating the EC2 server, starting Kafka in KRaft mode, creating the topic, connecting to AWS, and running everything. Don’t hardcode secrets.

Add simple error handling, readable logging, and troubleshooting notes for common problems like Kafka connection issues, S3 permissions, and missing AWS credentials. Look up current docs online if you need to.