Comparative study of database management systems for large-scale financial institution data storage

Posted: Jan 18, 2019

• Joseph Taylor
• Levi Wilson
• Logan Johnson

Abstract

This comprehensive study investigates the performance characteristics of modern database management systems when deployed in large-scale financial institution environments. Traditional comparative analyses have typically focused on technical benchmarks in isolation, neglecting the complex interplay between regulatory compliance requirements, real-time transaction processing demands, and long-term data retention policies that characterize financial data ecosystems. Our research introduces a novel multi-dimensional evaluation framework that simultaneously assesses technical performance, operational costs, and compliance readiness across relational, NoSQL, and NewSQL database architectures. We deployed representative systems from each category—PostgreSQL for relational, MongoDB for document-oriented NoSQL, and CockroachDB for NewSQL—in a simulated financial environment processing over 10 million transactions daily while maintaining compliance with financial regulations including Basel III, MiFID II, and GDPR. The methodology incorporates a unique stress-testing protocol that combines synthetic workload generation with real financial transaction patterns obtained through partnerships with participating institutions. Our findings reveal several counterintuitive results, including that NoSQL systems demonstrate superior performance for certain regulatory reporting tasks traditionally assumed to favor relational systems, and that the operational overhead of maintaining ACID compliance in distributed NewSQL environments varies non-linearly with transaction volume. The study concludes with a decision framework that enables financial institutions to select database technologies based on their specific balance of performance, compliance, and cost requirements, challenging several long-held assumptions in financial technology architecture.