The Market Data Platform Prototype is a cloud-based proof-of-concept (POC) that automates the collection, processing, and querying of commodities market data. Designed for real-time insights and efficient data retrieval, the system supports secure data storage, indexing, and user management across AWS services. 

The client wanted a cloud-based prototype to automate the flow of commodity market data. This included fetching data from an exchange, processing and storing it in MongoDB, saving it as JSON in Amazon S3, and indexing it in Amazon OpenSearch. User access to data was securely managed using Amazon Cognito.

Oodles delivered a cloud-native POC architecture, ensuring automation, scalability, and security:

• Automated Data Acquisition:
  Developed a Lambda function to fetch regularly and process commodities data from the exchange.
• Data Storage and Processing:
  Processed data was stored in a MongoDB cluster running in Docker containers for ease of deployment.
• JSON Archiving in Amazon S3:
  Transformed the processed data into structured JSON format and archived it in Amazon S3 for durability.

• Real-Time Indexing with OpenSearch:
  Created a secondary Lambda function to detect new S3 files, parse the JSON data, and index it into OpenSearch for real-time queries.