With the dawning of “Big Data” and Analytics, we as Software Engineers and Developers need to, more than ever, efficiently utilize all computing resources, both hardware and software, in order for our applications to optimally harvest and process such data.
This XPages Insights into Big Data OpenNTF project contains a “Phase 1” sample application that demonstrates two things:
– A simple data partitioning pattern to enable “Big Data” scalability (potentially TeraBytes ++). Achieved via a Managed Bean routing layer within a data-less XPages “dashboard” application.
– Multi-database parallel searching across a large data-set. When a search is requested within the UI, a multi-threaded concurrent search is carried out across twenty-six different databases before providing a seamless search result back to the end-user.
This is achieved via a Managed Bean that executes multi-threaded search logic using a special Java abstract class implementation that allows the current user session to be shared across multiple threads:
Note that this sample is not production ready – it’s a sample to provoke thought and learning – in fact it intentionally contains a number of coding and configuration “flaws” and/or omissions to limit the performance and scalability of the sample.
In the coming weeks a follow-up “Phase 2” release and video will be posted in conjunction with the XPages Masterclass Video Series – Series 2 to demonstrate and explain how to unleash the limits of this application for truly high-end performance and scalability in light of the trade-off’s between computing power and resources – that is JVM memory vs. disk memory vs CPU power vs high-end workload and functional expectations. It will also demonstrate augmentation of search results from Domino and RDBMS datasets and displaying those seamlessly to the end user. Stay tuned!