Market Data Platform Prototype
SUPPORT AND MAINTENANCE
Lancer:
<= 2 resources OR <=3 weeks effort OR <=$3000 USD
Growth:2 - 4 resources AND upto 4 months OR
upto $30000 USD
Enterprise:Everything else
(Development)
Project Description
The Market Data Platform Prototype is a cloud-based proof-of-concept (POC) that automates the collection, processing, and querying of commodities market data. Designed for real-time insights and efficient data retrieval, the system supports secure data storage, indexing, and user management across AWS services.
Scope Of Work
The client wanted a cloud-based prototype to automate the flow of commodity market data. This included fetching data from an exchange, processing and storing it in MongoDB, saving it as JSON in Amazon S3, and indexing it in Amazon OpenSearch. User access to data was securely managed using Amazon Cognito.
Our Solution
Oodles delivered a cloud-native POC architecture, ensuring automation, scalability, and security:
- Automated Data Acquisition:
Developed a Lambda function to fetch regularly and process commodities data from the exchange. - Data Storage and Processing:
Processed data was stored in a MongoDB cluster running in Docker containers for ease of deployment. - JSON Archiving in Amazon S3:
Transformed the processed data into structured JSON format and archived it in Amazon S3 for durability. Real-Time Indexing with OpenSearch:
Created a secondary Lambda function to detect new S3 files, parse the JSON data, and index it into OpenSearch for real-time queries.
Related Projects
Technologies Involved
Area Of Work
Technologies Involved
Area Of Work
