Work Experience

Designed and implemented a data processing platform in Spark on Kubernetes through reading multiple sources from AWS SQS and S3 for tiered log data ingestion and processing, transforming and enriching streaming data and storing processed data in Delta Lake tables.

Optimized Spark application performance through notification based source listing, data partitioning, broadcasting shared variables, and parallelizing Delta Lake table checkpoint writing.

Implemented Spark service management and deployment through Spinnaker pipelines and Airflow DAGs on Salesforce Hyperforce Cloud.

Designed and implemented a profiling system in Python for services and jobs running on Mesos cluster in EC2 through processing Mesos streaming events and service/job metrics, and loading data into Postgres.

Implemented a platform tool in ReactJS and Python for Mesos cluster monitoring and analysis through integrating data from various sources: Mesos, Aurora, Prometheus and Splunk.

Upgraded Mesos cluster to utilize oversubscribed revocable resources (cpu, ram), implemented a tool in Python for migrating Aurora services to use oversubscribed resources for increasing resource utilization.

Developed a system performance predictor in Python for Consul cluster workload prediction using Machine Learning algorithms such as SVM, Random Forest, Gradient Boosting Tree.

Implemented server buffer in Go to improve cluster stability and reduce response latency and failure.

Designed and implemented a project management system on Struts+Spring+Hibernate framework.

Implemented information retrieval and display using JSP, JavaScript and Ajax, and loaded project archives into MySQL database.

Implemented Java Servlets and filters for service actions (such as information update, remove) and security validation (such as user access control), applied Apache Tomcat as the web server in Linux.

Analyzed ANTLR (an open source parser generator) to learn C# parser and Script#.

Applied Script# to write JavaScript using C#.