A lightweight helper utility which allows developers to do interactive pipeline development by having a unified source code for both DLT run and Non-DLT interactive notebook run.
-
Updated
Dec 7, 2022 - Python
#
big-data-processing
Here are 25 public repositories matching this topic...
big data processing and machine learning platform，just like useing sql
-
Updated
Oct 15, 2024 - Python
A pipeline that consumes twitter data to extract meaningful insights about a variety of topics using the following technologies: twitter API, Kafka, MongoDB, and Tableau.
kafka big-data mongodb twitter-api data-visualization zookeeper data-analytics kafka-consumer kafka-producer tableau nosql-database kafka-streaming big-data-processing data-processing-pipelines
-
Updated
Aug 2, 2021 - Python
Implementation of algorithms for big data using python, numpy, pandas.
python bloom-filter lsh streams frequent-itemset-mining pcy frequent-itemsets stream-mining shingling big-data-processing lsh-algorithm min-hasing similar-items a-priori multistage-pcy multihash-pcy
-
Updated
Apr 27, 2020 - Python
From traffic sensors to smarter cities: real-time congestion prediction with Kafka, Spark, LSTM, XGBoost, and dynamic routing powered by graph algorithms.
visualization machine-learning kafka big-data spark algorithms data-engineering optimization-algorithms streaming-data big-data-processing streamlit
-
Updated
Sep 25, 2025 - Python
The following readme file, assume that before running the Spark analytic job, you have already installed the correct versions of **Java**, **Hadoop**, **Spark** and that you are inside **Ubuntu**.
java ubuntu distributed-computing open-data batch-processing data-pipeline hadoop-mapreduce cluster-computing linux-environment big-data-processing apache-sparksql data-analytics-project
-
Updated
May 8, 2025 - Python
rock-solid pillars for enterprise-grade solutions
python vue jupyter ubuntu rethinkdb rhel rust-lang nginx-unit tailwindcss big-data-processing py-script
-
Updated
Feb 5, 2024 - Python
excel, markdown, csv, sql 数据源批量/单独格式互相转换
easy-to-use data-preprocessing etl-framework big-data-processing csv-to-excel csv-to-sql multifileupload data-cleaning-pipeline excel-to-md
-
Updated
Nov 23, 2023 - Python
Simple CSV parser for huge volumes of data with the use of the library Pandas for Python for getting specific columns of a CSV file and putting the extracted data into one or more files (each column in a separated file or all of them in the same output) in a short amount of time.
parser csv data-manipulation pandas-dataframes conda-environment pandas-datareader big-data-processing
-
Updated
Jan 7, 2019 - Python
Sentiment-Analysis-API
nlp machine-learning deep-learning sentiment-analysis neural-network lstm-neural-networks rnn-pytorch big-data-processing
-
Updated
Jul 18, 2022 - Python
Building Data Lake and ETL pipelines using Amazon EMR, S3, and Apache Spark
aws sql big-data spark amazon-emr pyspark data-engineering data-lake cloud-computing amazon-s3 etl-pipeline big-data-processing
-
Updated
Dec 23, 2022 - Python
BigQuery data pipeline with dbt, Spark, Docker, Airflow, Terraform, GCP
python docker bigquery airflow spark terraform pyspark dbt elt batch-processing big-data-analytics etl-pipeline big-data-processing elt-pipeline
-
Updated
Feb 6, 2023 - Python
Setting up a Spark cluster in a Docker environment for improved repeatability and reliability. This project includes a simple transformation on a dataset containing approximately 31 million rows.
-
Updated
Jun 21, 2024 - Python
Hands-on project demos covering infrastructure automation (Ansible, Docker), big-data processing & streaming (Hive, Spark, Kafka), and network experiments (MitM, TCP-over-UDP).
docker tmux ansible devops spark networking hive docker-compose hbase spark-streaming dataops mapreduce kafka-producer hadoop-mapreduce spark-sql reliable-udp hiveql mapreduce-python big-data-processing mitm-attack
-
Updated
Feb 20, 2026 - Python
Kappa Architecture Based Sentiment Analysis System for User Comments
-
Updated
Feb 10, 2025 - Python
A practical coursework-style project from my Master's studies in Big Data Analytics (at University of East London), showcasing hands-on use of big data tools and techniques on a real-world cyber-security dataset.
machine-learning sql big-data spark hive hadoop distributed-computing pyspark data-engineering cybersecurity intrusion-detection hdfs data-analysis network-security spark-mllib big-data-analytics hiveql big-data-processing big-data-engineering unsw-nb15-dataset
-
Updated
Nov 18, 2025 - Python
End-to-end big data pipeline for delivery operations analytics using distributed storage, batch & stream processing, and live dashboards to support operational monitoring and decision-making.
-
Updated
Jun 3, 2026 - Python
Exploring and Implementing Scalable Data Processing Techniques
big-data spark multiprocessing multithreading pyspark word-association big-data-analytics big-data-processing krager-mincut
-
Updated
May 11, 2023 - Python
This work is from my master thesis: Condition Monitoring with Machine Learning: A Data-Driven Framework for Quantifying Wind Turbine Energy Loss.
machine-learning big-data wind-turbine hawt big-data-analytics condition-monitoring big-data-processing energy-loss
-
Updated
Jun 21, 2025 - Python
Solved tasks of the master's degree courses of speciality "Algorithms and Systems for Big Data Processing".
machine-learning information-retrieval highload mpi neural-networks external-memory university-course python-course randomized-algorithms cache-optimization page-rank big-data-processing
-
Updated
Jan 13, 2019 - Python
Improve this page
Add a description, image, and links to the big-data-processing topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the big-data-processing topic, visit your repo's landing page and select "manage topics."