Velan Masilamani A.

This project is to develop a Cloud AI based Automated Loan Enquiry & Application processing platform in Real Time which can Generate a Loan Proposal document and share it with Loan Applicant in quickest possible time to their registered mail box & WhatsApp messaging App enabling Real time processing & decision making on Loan Applications. In a Medallion/Microservices Architecture, Kafka topics is used to ingest the incoming loan enquiry & application real time data from different data sources and Databricks Real time Streaming pipeline jobs will consume these data from Kafka for performing Data Transformation using Spark libraries. Data Lakehouse is used to store the structured & semi-structured processed data in Delta table format for AI programs to perform decision processing in real time. For faster insights, processed & high quality pre-aggregated data is loaded into Snowflake for generating Loan Proposal document to be shared with Loan Applicant and also for external 3rd party data consumption thru' Reporting & Analytics Tools.

My Contributions as Lead Data Engineer/Data Architect

•Architected End to End Solution based on Medallion and Microservices Architecture principles.

•Designed and Developed Real time data ingestion, streaming, processing and transformation pipelines using Kafka topics and Databricks Spark Notebook.

•Created a Data model for both Data Lakehouse using Data Vault Modelling and Snowflake storage.

•Involved in performance optimization of Databricks and Snowflake to speed up the processing time with minimal computing & resources cost.

•Collaborated with Business Analyst for Requirements analysis, QA team for real time test automation approach and Devops for deployments.

•Mentored and guided Developers on coding design and compliance to best practices, standards & principles. Coordinated with Business Teams on Solution Design and UAT support.

• • Analyze, design and develop Azure cloud based data solutions to support Data Ingestion, Extraction, Transformation, Load and Visualization of Policies & Claims data.

• • Developing ETL programs for Real Time and Batch data processing of Policy & Claims from multiple insurance data source systems ("golden source") to Azure Data Storage services using Azure Data Factory pipelines, Pyspark, Databricks Notebook, Spark SQL, Data Lake, Blob Storage & Azure SQL.

• • Created pipelines for batch processing in ADF & real time data integration in Databricks with different source data formats like SQL Server, CSV, Parquet, XML, JSON, Salesforce API, Blob storages, etc

• • Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation and aggregation from multiple insurance data formats for analyzing and transforming insurance policy & claims data.

• • Responsible for design & build of Reconciliation code for policy premiums and commissions.

• • Data integration with Salesforce CRM and pushing the data back thru Salesforce API using ADF Linked Services.

• • Data integration with Master Data Management systems for Industry Classification Codes, Geography zip codes, Catastrophe codes, etc

• • Supported data integration POC solutions for evaluating new third party insurance products.

• • Involved in Data Governance, Data Catalog & Data Lineage.

• • Mentoring & guiding the new developers and performing technical management of the development tasks.

• • Adherence to best practices and data model principles of Data Vault Modeling, Dimension Modeling, SCD Type 2, Star Schema & Snowflake Schema.

• • Implemented SAFe Agile Release Train (ART) with 7-10 Agile Scrum Teams (around 70-80 members) and collaborated with Release Train Engineers.