Professional for Hire
I love building data products that scale. That means implementing simple solutions with minimal maintenance through automation and eloquent designs.
My software experience spans the full stack; from system level deployment to application implementation. In particular, I spent quite a bit of time working with the technologies
- Apache Spark (Spark Streaming, Spark SQL)
- Amazon Web Services (Kinesis, Redshift, S3, EC2, Route53, VPC)
- Python (Flask, Gunicorn)
- Scala (Play, sbt)
A few projects that I designed, implemented and delivered to a production environment.
Data pipeline and analytics framework
Implemented a high-throughput data pipeline that enabled massive amounts of data to be published to AWS Kinesis and consumed with Apache Spark. The infrastructure enabled in-flight computation as well as data integration with a variety of datastores.
- Data pushed to S3, AWS Redshift and MongoDB within 1 minute.
- Supports real-time machine learning and aggregation.
Built a highly scalable analytics reporting system as part of the core product of our software as a service (SaaS) platform. The system was designed using a services oriented architecture (SOA) model to minimize coupling. It also supported horizontal scaling by distributing writes and reads across nodes. Over a million data points can be processed within minutes and rendered within seconds to the end user.
- Dashboard metrics are updated within a minute.
- Supports up to ~500 million events per day.
Cloud infrastructure and automation
Automated the deployment and buildup of an AWS cloud infrastructure that auto-scales with traffic demand. The infrastructure was automated using Ansible and enabled the deployment of multi-instance clusters with varying configuration and topology.
- Multi-instance cluster collapsed into a single-instance.
- Tear-down or build-up an environment within 5 minutes.
- Decouple subsystems with independent deployment trains.
Decoupled platform architecture
Architected as well as implemented a new configuration management system for the core QP platform at QuickMobile. The design decoupled the system services from the application to enable greater flexibility and reduced down-down during deployments. The configuration mechanism was implemented using Ansible in idempotent style.