More on HazelcastΒΆ
Hazelcast is a JAVA framework to manage in-memory data grid. The main features provided by Hazelcast are:
In-Memory Data Grid (IMDG): Hazelcast offers a distributed in-memory data grid that allows you to store and access data in RAM across multiple nodes in a cluster. This enables fast and low-latency access to data, improving overall application performance.
Distributed Data Structures: Hazelcast provides various distributed data structures like maps, lists, sets, queues, and locks. These data structures are distributed across the cluster and can be accessed and manipulated concurrently by multiple applications or threads.
Distributed Computing: Hazelcast allows you to perform distributed computing tasks by distributing the workload across multiple nodes in a cluster. It supports parallel processing, task distribution, and result aggregation, enabling efficient and scalable computation.
Caching: Hazelcast offers a distributed caching mechanism that helps improve application performance by caching frequently accessed data in memory. It provides support for both near-cache and distributed cache configurations.
Eventual Consistency and Replication: Hazelcast ensures eventual consistency by replicating data across multiple nodes in a cluster. This replication provides fault tolerance and high availability, ensuring that data is not lost in the event of a node failure.
Distributed Messaging: Hazelcast supports distributed messaging patterns, allowing applications to communicate with each other through publish-subscribe and point-to-point messaging. It provides reliable and asynchronous messaging between distributed components.
Distributed Query and Indexing: Hazelcast allows you to query and index data stored in distributed data structures. It supports various query languages and indexing techniques, enabling efficient and fast data retrieval.
Integration with Java and Other Languages: Hazelcast provides client libraries and APIs for Java, .NET, C++, Python, and other languages, making it easy to integrate Hazelcast with existing applications and systems.
Overall, Hazelcast offers a comprehensive set of features for distributed data management, caching, computing, messaging, and querying, enabling developers to build scalable and high-performance applications.
-------------------
| Client | (Genius Server Relay)
-------------------
| | |
| | |
v v v
-------------------
| Hazelcast | (Genius Server Node 1...n) = Cluster
| Cluster/Node |
-------------------
| | |
| | |
v v v
-------------------
| Data Storage |
| (In-Memory) |
-------------------
In the above diagram you can see that clients directly interact with the Hazelcast cluster, using client libraries specific to the used programming language. The client is in this example the Genius Relay.
The Hazelcast cluster (implemented by 1 or more Genius Server nodes) consists of multiple nodes, which can be distributed across different machines. Nodes communicate with each other and form a cohesive cluster.
The Hazelcast cluster stores data in distributed in-memory storage. Data can be organized into data structures such as maps, lists, sets, queues, etc. Each node in the cluster holds a portion of the distributed data. Data distribution happens automatically and transparently to the application.
Clients can perform operations on the data using Hazelcast client libraries. Operations can include reads, writes, updates, and queries. Hazelcast handles synchronization and consistency of data across cluster nodes. In case of a node failure, data is automatically restored and kept available. The Hazelcast cluster also supports distributed messaging capabilities, allowing clients to send messages to other components within the cluster.
In our case the hazelcast infrastructure it's main focused in this points:
Relay commands: several commands are implemented by the relay that are jobs called from the Genius Server synchronously and are local tasks executed by the relay. For example deleting an orphaned file:
----------------------- ask for Relay Command -----------------
| Genius Server | ======================> | Relay execute |
| Node | Response | command local |
----------------------- <===================== -----------------
Server executor: similar to the Relay commands, but they are called from the Genius Relay to the Genius Server, and they can be synchronous but don't have to be.
Task with answer:
-------------- ask for Genius Server task --------------------------
| Relay | ========================> | Genius Server node |
| Node | Response callable | command local |
-------------- <======================= --------------------------
Task async:
-------------- ask for Genius Server task --------------------------
| Relay | =========================> | Genius Server node |
| Node | Asynchronous | command local |
-------------- --------------------------
Every communication between the relay and the server nodes are done with serializable beans (DTO) through the TCP communication port set from configuration (default 5701).
Hazelcast also plays an important role with regard to dynamic configuration. At the moment every time we change something in the dynamic configurations (/conf) the change is replicated in all nodes of the cluster, always through listeners.
The Hazelcast shared map is also used for initialization of SnmpV3 users, any changes or additions, must be synchronized with all nodes of the cluster and relays. The component that manages SnmpV3 needs a user registry, without which it would not work. When this registry is updated all the changes are put on a map that has a registered listener which allows all nodes registered in the Hazelcast cluster to receive the new log. If a node was offline, the registry would still be loaded when it restarted.
If the architecture includes at least one relay or more than one Genius Server node, Hazelcast is mandatory.