Fuse is a framework for creating distributed applications and as such it has many moving parts. There are bundles, features, profiles, containers and various ways of combining those into the resulting deployment architecture. Here are some thoughts and guidelines based on the field experiences from a number of projects.
If we start from bottom up, we have bundles, features and profiles as the main resulting artifacts from the development process.
An OSGI bundle corresponds to a maven module and it is basically a jar file with additional MANIFEST.MF that contains OSGI specific metadata. A developer doesn't have to do much other than declare the maven packaging type as a bundle and let maven-bundle-plugin generate MANIFEST.MF file at compile time.
A Karaf feature is the preferred way for declaring dependencies in Karaf. So rather than installing each bundle and its runtime dependencies one by one, features allow us to group declaratively a bunch of bundles that should be deployed together at run time. Think of features as a way to describe the minimal deployment unit that represents a business capability (kinda a miroservice descriptor ;)
An example may help here: let's say we have an order-domain bundle, order-spec bundle with a wsdl and xml schemas, order-routes bundle with few Camel routes, and order-persistence bundle. And order-feature would combine all these bundles and other dependencies (which in turn can be bundles or features) such as camel components and third party libraries. The point is that order-feature is the minimal deployment unit that provides business value and represents a runnable unit of code (with inputs and outputs). It is also possible to use features just to group a set of bundles that usually go together and provide technical capability like JTA/JPA feature. Having a business capability and all of its dependencies declared in one place makes it really easy to move that deployment unit from container to container for scalability and HA purposes. Deploying and running features to a container is achieved through profiles.
Fuse profiles consist of two main ingredients: binaries (which are bundles described by features) and configurations/properties (which are usually environment specific) needed to run the binaries. So I would create an order-profile (that maps 1 to 1 to order-feature) containing order-feature and set of property files (to configure order-persistence bundle with DB configs, order-routes bundle with port number of the order service, etc). So order-profile would contain the same binaries on all environments but different property files (specific for each environment - dev/test/int/prod).
So we have camel routes->bundles->features->profiles. The next question is how do we organize profiles into containers considering that in addition to the business profiles we create during development, there are also out-of-box technical profiles that a typical Fuse application uses. Let's clarify the term container first. A container is overloaded term, especially with the popularity of Docker containers, but when we say container in the Fuse world, it refers to Karaf container, that is a standalone Java process (imagine a microservice instance if you prefer). And the big magic done by Fuse Fabric is the ability to deploy and run code on Karaf containers by assigning profiles to a container. So profiles dictates what features and bundles are deployed on certain container giving a container a specific responsibility/flavor.
Next, let's have a look at different types of containers we have in a typical Fuse application:
So we have our containers deployed with profiles which in turn assign specific responsibilities to the containers. How do we distribute these containers into available nodes/hosts(physical or VMs)? This is a very common question and the answer is as you may have guesses: it dependents. It dependents on the available resources and non functional requirements. The main consideration at this stage is to avoid single point of failure, enable horizontal scaling of the application based on the anticipated load and keeping in mind application maintainability too. All this sounds nice but doesn't say anything concrete. So here are few areas to think further before making a decision and connecting the boxes in the deployment diagram. Each of these different types of Karaf containers has specific resource and availability requirements and depending on the expected load and available resource you will end up choosing one or another way of deploying these profiles.
Fabric Servers require odd number of instances, with minim of 3 but preferably 5 or more. That is because ZK favors consistency over availability and it requires the majority if its instances be up and running. If that condition is not met, the rest of the ZK instances stop functioning, meaning the other containers will not have any of the Fabric services such as A-MQ discovery, CXF-discovery, master-component, provisioning available. To prevent that, it is recommended to isolate the Fabric Server containers into separate nodes with good connectivity between the nodes and not run other containers on the same host that may steal resources. These containers don't require too much resources, so probably 1-2 cores and 1-2GB of heap will be enough in most cases but they are essential for the rest of the Fabric!
The other fundamental part of a Fabric installation are the A-MQ containers. If the application is using messaging layer and the broker goes down, the rest of the system becomes useless, that's why you would usually have A-MQ either in Master-Slave or/and in Network of Brokers cluster. In most cases that will be Master-Slave setup, where the two broker containers are split in separate hosts with access to shared datastore. In such a setup you will have an active A-MQ instance on one host and passive A-MQ instance in the second host. As such the second host will not be utilized at all, and it is good idea to put another set of Master-Slave cluster in the opposite hosts to have better utilization of the resources. Also keep in mind the broker resource appetite (cpu/memory/io) is directly dependent on the message load, so treat your broker with the same respect as you would treat a database. If the broker is persistent, it will be IO intensive, if there is not enough memory the broker will flush to disk, if your consumers are slow you may run out of disk space, if there is not enough processing power, you may run out of heap space... Many ways to kill a broker, if it doesn't have the resources it needs.
The containers that require some more imagination in terms of deployment topology are the ESB/Worker containers. The way these containers are deployed is really dependent on how many business services are running, what are the dependencies between those services and what are the non functional requirements for each service. A service (usually consisting of few Camel routes) may be CPU intensive (complex calculation for example) or memory intensive (huge messages) or IO intensive (typically file copying) or a combination of those. A service might be a low latency real time service with auto scaling requirements, or business critical that should be highly available, or batch oriented with manual fail over, etc. So when deploying multiple business profiles into one container or multiple containers into one host consider the following options for each service:
Monitoring containers with Fabric Insight profiles are new to Fuse and their role is primarily running Elastic Search servers that read and write logs to Lucene indexes. These containers will be primarily IO intensive and not critical for the rest of system, but keep in mind the available disk space and how much logs you would want to keep around. From business perspective it might be hard to justify dedicated hosts only for logging purposes, so you might end up sticking monitoring containers on Fabric Server hosts (makes sense as both types of containers have monitoring/management responsibilities) or to any host that have some additional capacity.
Fabric Gateway is a lightweight load balancing and discovery service for CXF and A-MQ endpoints that are managed by Fabric. As this profile has a distinct and crucial responsibility that enables consumers to discover service providers, it seems logical to have multiple instances of such containers located on the hosts with the service providers or any other hosts including locating them on the host where the consumer is.
Below is a diagram that illustrates a sample Fuse deployment topology following the above thoughts. Depending on your use case, you may have more or less Fabric servers (hopefully 5 rather than 1), may be only one A-MQ Master/Slave cluster, and more ESB containers. If you can get also connectivity from build server to the Fabric Server 1, that will enable automatic deployments (requires some scripting) and have full CI/CD cycles (also your Fabric servers will need access to maven repositories).
No matter whether you call it SOA or Microsservices, ESB or light integration layer, distributed applications have many moving parts and building one is a complex (and enjoyable) journey. The post here is quite high level and focusing on the happy paths only. For any real discussions, have a chat with Red Hat.
If we start from bottom up, we have bundles, features and profiles as the main resulting artifacts from the development process.
An OSGI bundle corresponds to a maven module and it is basically a jar file with additional MANIFEST.MF that contains OSGI specific metadata. A developer doesn't have to do much other than declare the maven packaging type as a bundle and let maven-bundle-plugin generate MANIFEST.MF file at compile time.
A Karaf feature is the preferred way for declaring dependencies in Karaf. So rather than installing each bundle and its runtime dependencies one by one, features allow us to group declaratively a bunch of bundles that should be deployed together at run time. Think of features as a way to describe the minimal deployment unit that represents a business capability (kinda a miroservice descriptor ;)
An example may help here: let's say we have an order-domain bundle, order-spec bundle with a wsdl and xml schemas, order-routes bundle with few Camel routes, and order-persistence bundle. And order-feature would combine all these bundles and other dependencies (which in turn can be bundles or features) such as camel components and third party libraries. The point is that order-feature is the minimal deployment unit that provides business value and represents a runnable unit of code (with inputs and outputs). It is also possible to use features just to group a set of bundles that usually go together and provide technical capability like JTA/JPA feature. Having a business capability and all of its dependencies declared in one place makes it really easy to move that deployment unit from container to container for scalability and HA purposes. Deploying and running features to a container is achieved through profiles.
Fuse profiles consist of two main ingredients: binaries (which are bundles described by features) and configurations/properties (which are usually environment specific) needed to run the binaries. So I would create an order-profile (that maps 1 to 1 to order-feature) containing order-feature and set of property files (to configure order-persistence bundle with DB configs, order-routes bundle with port number of the order service, etc). So order-profile would contain the same binaries on all environments but different property files (specific for each environment - dev/test/int/prod).
So we have camel routes->bundles->features->profiles. The next question is how do we organize profiles into containers considering that in addition to the business profiles we create during development, there are also out-of-box technical profiles that a typical Fuse application uses. Let's clarify the term container first. A container is overloaded term, especially with the popularity of Docker containers, but when we say container in the Fuse world, it refers to Karaf container, that is a standalone Java process (imagine a microservice instance if you prefer). And the big magic done by Fuse Fabric is the ability to deploy and run code on Karaf containers by assigning profiles to a container. So profiles dictates what features and bundles are deployed on certain container giving a container a specific responsibility/flavor.
Next, let's have a look at different types of containers we have in a typical Fuse application:
- Fabric Server - This is where the ZooKeeper server is running, together with the maven proxy, git server and Hawtio instance.
- A-MQ Container - runs A-MQ server instance(s).
- Worker/ESB container - this is where the business/integration logic is running. That is typically Camel and CXF based (our order profile would be deployed here).
- Insight container - is where the Elastic Search server runs.
- Fabric Gateway - used for discovery and load balancing of CXF or A-MQ endpoints.
So we have our containers deployed with profiles which in turn assign specific responsibilities to the containers. How do we distribute these containers into available nodes/hosts(physical or VMs)? This is a very common question and the answer is as you may have guesses: it dependents. It dependents on the available resources and non functional requirements. The main consideration at this stage is to avoid single point of failure, enable horizontal scaling of the application based on the anticipated load and keeping in mind application maintainability too. All this sounds nice but doesn't say anything concrete. So here are few areas to think further before making a decision and connecting the boxes in the deployment diagram. Each of these different types of Karaf containers has specific resource and availability requirements and depending on the expected load and available resource you will end up choosing one or another way of deploying these profiles.
Fabric Servers require odd number of instances, with minim of 3 but preferably 5 or more. That is because ZK favors consistency over availability and it requires the majority if its instances be up and running. If that condition is not met, the rest of the ZK instances stop functioning, meaning the other containers will not have any of the Fabric services such as A-MQ discovery, CXF-discovery, master-component, provisioning available. To prevent that, it is recommended to isolate the Fabric Server containers into separate nodes with good connectivity between the nodes and not run other containers on the same host that may steal resources. These containers don't require too much resources, so probably 1-2 cores and 1-2GB of heap will be enough in most cases but they are essential for the rest of the Fabric!
The other fundamental part of a Fabric installation are the A-MQ containers. If the application is using messaging layer and the broker goes down, the rest of the system becomes useless, that's why you would usually have A-MQ either in Master-Slave or/and in Network of Brokers cluster. In most cases that will be Master-Slave setup, where the two broker containers are split in separate hosts with access to shared datastore. In such a setup you will have an active A-MQ instance on one host and passive A-MQ instance in the second host. As such the second host will not be utilized at all, and it is good idea to put another set of Master-Slave cluster in the opposite hosts to have better utilization of the resources. Also keep in mind the broker resource appetite (cpu/memory/io) is directly dependent on the message load, so treat your broker with the same respect as you would treat a database. If the broker is persistent, it will be IO intensive, if there is not enough memory the broker will flush to disk, if your consumers are slow you may run out of disk space, if there is not enough processing power, you may run out of heap space... Many ways to kill a broker, if it doesn't have the resources it needs.
The containers that require some more imagination in terms of deployment topology are the ESB/Worker containers. The way these containers are deployed is really dependent on how many business services are running, what are the dependencies between those services and what are the non functional requirements for each service. A service (usually consisting of few Camel routes) may be CPU intensive (complex calculation for example) or memory intensive (huge messages) or IO intensive (typically file copying) or a combination of those. A service might be a low latency real time service with auto scaling requirements, or business critical that should be highly available, or batch oriented with manual fail over, etc. So when deploying multiple business profiles into one container or multiple containers into one host consider the following options for each service:
- Business criticality: highly critical services on dedicated containers and safest hosts
- Load: services with high load on hosts with more resources
- Coupling: services coupled together (direct in memory calls are more efficient that http or jms for service interaction) or service depending on same resources
- Functionality: services belonging to the same business domain or applications layer
Monitoring containers with Fabric Insight profiles are new to Fuse and their role is primarily running Elastic Search servers that read and write logs to Lucene indexes. These containers will be primarily IO intensive and not critical for the rest of system, but keep in mind the available disk space and how much logs you would want to keep around. From business perspective it might be hard to justify dedicated hosts only for logging purposes, so you might end up sticking monitoring containers on Fabric Server hosts (makes sense as both types of containers have monitoring/management responsibilities) or to any host that have some additional capacity.
Fabric Gateway is a lightweight load balancing and discovery service for CXF and A-MQ endpoints that are managed by Fabric. As this profile has a distinct and crucial responsibility that enables consumers to discover service providers, it seems logical to have multiple instances of such containers located on the hosts with the service providers or any other hosts including locating them on the host where the consumer is.
Below is a diagram that illustrates a sample Fuse deployment topology following the above thoughts. Depending on your use case, you may have more or less Fabric servers (hopefully 5 rather than 1), may be only one A-MQ Master/Slave cluster, and more ESB containers. If you can get also connectivity from build server to the Fabric Server 1, that will enable automatic deployments (requires some scripting) and have full CI/CD cycles (also your Fabric servers will need access to maven repositories).
