About Arjuna Pandian
Senior Data Engineer and Data Platform Architect with 9+ years of experience building high-performance, cloud-native data ecosystems across hedge funds, banking, telecom, and insurance domains. Proven expertise in architecting low-latency streaming systems, large-scale batch processing frameworks, and enterprise lakehouse platforms using Spark, Flink, Kafka, Iceberg, and Snowflake.
Specialized in designing scalable data platforms for high-frequency trading and real-time analytics, enabling rapid research iteration, strong data governance, and production-grade reliability. Strong background in multi-cloud architecture (AWS & Azure), Kubernetes-based infrastructure, CI/CD automation, and embedded data quality frameworks using Great Expectations and dbt.
Recognized for leading greenfield platform builds, mentoring engineering teams, driving cost optimization (FinOps), and delivering resilient, high-throughput data systems that power business-critical decision-making.
English
Native or bilingual
Can work on-site
London (up to 50km)
Experience
- Accolite Labs Europe LimitedSystematic Trading - Data PlatformOctober 2023 - Today (2 years and 8 months)London, UK• • Architected and built a greenfield, scalable, cloud-native enterprise data platform to support high-frequency and systematic trading strategies for a hedge fund environment using PySpark, Pandas, AWS services and Iceberg for large-scale data feeds.• • Designed and implemented real-time and batch data ingestion pipelines using Apache Kafka, Kafka Connectors, and Apache Airflow, enabling low-latency access to market and alternative datasets.• • Leveraged Databricks notebooks and jobs for exploratory data analysis, data validation, and scalable transformations, enabling rapid iteration from local analysis to distribute production pipelines.• • Established best practices for data modelling, partitioning, and schema evolution with Iceberg and Trino for efficient querying.• • Integrated Snowflake as a cloud data warehouse for curated and analytics-ready datasets, implementing secure data sharing, role-based access control, and optimized warehouse sizing to support low-latency analytics and reporting.• • Automated infrastructure and CI/CD workflows using Terraform, Docker, and Kubernetes (EKS) across AWS and Azure.• • Implemented notebook-driven analytics and ad-hoc research workflows in Databricks, enabling quants and analysts to iteraterapidly using Pandas, Spark SQL, Snowflake and version-controlled pipelines.• • Deployed scalable, secure data services and automated releases via Azure DevOps and Jenkins.• • Developed real-time analytics dashboards with Dash Enterprise, integrating live market data for trading insights.• • Managed a team of developers and interns, taking full ownership of project delivery, mentoring junior team members, anddriving technical decisions to ensure high-quality, scalable solutions.
- Accolite Labs Europe LimitedDataOps EngineerMay 2023 - September 2023 (4 months)London, UK• • Improved data operations by automating workflows with Python, Microservices, and SQL, boosting processing efficiency andreducing manual intervention across telecom data systems.• • Built and managed secure RESTful APIs with FastAPI to provide controlled access to data for internal and external partners.• • Monitored and optimized real-time data pipelines using Kafka and Dynatrace, ensuring reliability and quick issue resolution.• • Strengthened data governance using audit logging, access controls, and validation checks on AWS and Linux environments.• • Collaborated with engineering and business teams to drive AWS cloud cost optimization (FinOps) practices, leveraging AWS Cost Explorer, Budgets, and tagging strategies to improve cost transparency and optimize cloud usage.
- Accolite Labs Europe LimitedSSE / Data EngineerJune 2019 - February 2022 (2 years and 8 months)Chennai, Tamil Nadu, India• • Architected and implemented scalable data pipelines using Python, Java, and Apache Kafka, improving data processing efficiency and enabling real-time analytics for business decision-making across backend systems.• • Led the migration from a legacy SQL-based data warehouse to a cloud-native data lake on AWS and Azure, ensuring dataintegrity and improving system performance for more efficient data retrieval and storage.• • Developed and integrated data-centric solutions including REST APIs, microservices (Spring Boot), and Fast API, ensuring seamless data flow and interoperability between systems while mentoring the team on best practices.• • Leveraged Low-Code/No-Code platforms (Unqork) to rapidly build data-driven applications, aligning with businessrequirements, and effectively translating technical specifications for non-technical stakeholders.
Recommendations
Be the first to recommend Arjuna Pandian
Help this freelancer shine by sharing your experience working together.
These freelancer profiles also match your criteria
Agatha Frydrych
Backend Java Software Engineer
4.7
(3)
2
Baptiste Duhen
Fullstack developer
4.6
(4)
5
Amed Hamou
Senior Lead Developer
4
(2)
7
Audrey Champion
Web developer
4.3
(3)
4
Education
- Master of TechnologyBirla Institute of Technology and Science2023Master of Technology
- Bachelor of TechnologyArunai Engineering College, Anna University2016Bachelor of Technology