Machine Learning Engineer
Stealth Startup
Job Overview
We are seeking a highly skilled Machine Learning Engineer with expertise in Large Language Models (LLMs) and AWS infrastructure to join our team. You will be responsible for building a robust data pipeline that ingests financial documents, performing data transformations, and training and fine-tuning LLMs for retrieval-augmented generation (RAG).
Key Responsibilities:
-
Data Ingestion & Processing: Build and maintain a data pipeline to retrieve, clean, and structure data using AWS services (Lambda, S3, Glue, etc.).
-
LLM Training & Fine-Tuning: Train, fine-tune, and optimize Large Language Models (LLMs) like LLaMA using AWS SageMaker and implement retrieval-augmented generation (RAG) techniques for improved accuracy.
-
AWS Architecture: Design and implement scalable AWS infrastructure for data storage, ETL processes, model training, and inference (S3, SageMaker, Glue, Athena, OpenSearch, etc.).
-
API & Frontend Interface Development: Build and integrate REST APIs (using AWS API Gateway and Lambda) that connect the LLMs with a frontend interface (AWS Amplify/CloudFront).
-
Search and Retrieval: Implement search functionality using AWS OpenSearch for document indexing and retrieval to support LLM fine-tuning and generate accurate, contextualized responses.
-
Optimization: Work on performance tuning and cost optimization across AWS services to ensure high availability, low latency, and efficiency.
Required Qualifications:
-
Strong experience with AWS services, including Lambda, S3, SageMaker, Glue, Athena, and OpenSearch.
-
Hands-on experience with model training, deployment, and inference in cloud environments (preferably AWS SageMaker).
-
Proficiency in Python, including data processing and model training libraries (PyTorch, TensorFlow, Hugging Face, etc.).
-
Experience working with financial data and/or document retrieval systems is a plus. Strong understanding of ETL processes and building scalable data pipelines.
-
Excellent problem-solving skills, and ability to work independently and collaboratively in a fast-paced environment.
Preferred Qualifications:
-
Experience with retrieval-augmented generation (RAG) techniques.
-
Familiarity with deploying user interfaces on AWS (Amplify, CloudFront). ● Knowledge of financial markets, SEC filings, and investment data is highly advantageous.