Repository

https://github.com/UnionVMS/UVMS-MDRCacheModule-APP 
QA Summary App: https://sonarcloud.io/dashboard?id=eu.europa.ec.fisheries.uvms.mdr%3Amdr
CI location App: https://jenkins.focus.fish/view/UVMS%20App/job/UVMS-MDRCache-APP/
Project Site reports App: https://jenkins.focus.fish/view/Site-Reports/job/UVMS-MDR-APP-Site-Reports/site/
Tattletale report: https://jenkins.focus.fish/view/Site-Reports/job/UVMS-MDR-APP-Site-Reports/ws/deploy-wildfly/target/site/tattleTaleReport/index.html

 Description

MDR module is a caching mechanism which synchronizes/refreshes data (code lists) fetched from the MDM (EC MDR Module).

The reason for the existence of such a module is that if we fetched the data directly from MDM (during normal application navigation) the application that uses those data (Ex. Activity module) and the web interface would hang each time for minutes.

This module offers web services to synchronize particular ones or all code lists and to retrieve the desired ones by search, a mechanism backed by Lucene indexing for faster fetching.

It also exposes the code lists content through JMS communication so that other modules can fetch data from it and use them in their back-end logic.

To work correctly and in order for the synchronization features to be available the mdr-plugin need to be deployed.

So here are all the dependencies that are need for the Mdr module to work correctly as a whole : 

1. Mdr App is the component that provides all the Web-Services, Jms Services, and interactions with the DB layer.

      2. Mdr Model provides the classes that are used for communication purposes between differend modules and the front-end, meaning: DTOs, XSDs, Mappers, JAXBMarshallers etc.                              

      3. MDR DB is a “utility” module (To be merged with APP later on) which is used mainly to store the Liquibase format instruction for recreating the MDR Schema. Within this module comes also the test data.

      4. The MDR FLUX Plugin is an application that forwards the request messages (related to the code lists) to the next node (In our case the AHR:VMS node) which in the end is dispatched to the EC MDR.

          I also does the opposite, meaning receiving and sending the EC MDR responses to be saved in our MDR Cache module.    

Module dependencies

JMS queue dependencies

Web Services

The module offers the below enlisted web services.

The request should be issued in the following format :  {{hostname}}:{{port}}/mdr/rest/{{ path_to_specific_service }}

In this table only the {{path_to_specific_service}} will be shown.

On top of this there is also an automatic scheduler which automatically fetches all the data from MDM and refreshes MDR DB. This scheduler uses a cron-expression to determine how frequently to update the code-lists. The scheduler is configurable  by the user. It takes into consideration only lists for which the configurable flas is set to true.

Datasources

<xa-datasource jta="true" jndi-name="java:jboss/datasources/uvms_mdr" pool-name="uvms_mdr" enabled="true" use-ccm="true" statistics-enabled="true">
    <xa-datasource-property name="ServerName">postgres</xa-datasource-property>
    <xa-datasource-property name="PortNumber">5432</xa-datasource-property>
    <xa-datasource-property name="DatabaseName">db71u</xa-datasource-property>
    <driver>postgresql</driver>
    <xa-pool>
        <min-pool-size>2</min-pool-size>
        <initial-pool-size>2</initial-pool-size>
        <max-pool-size>20</max-pool-size>
        <prefill>true</prefill>
    </xa-pool>
    <security>
        <user-name>mdr</user-name>
        <password>mdr</password>
    </security>
</xa-datasource>

Liquibase Scripts

https://github.com/UnionVMS/UVMS-MDRCacheModule-DB/tree/master/LIQUIBASE

Related Repositories

https://github.com/UnionVMS/UVMS-MDRCacheModule-MODEL

Jenkins Builds

https://jenkins.focus.fish/job/UVMS-MDRCache-APP/

https://jenkins.focus.fish/job/UVMS-MDRCache-MODEL/

JMS Services

This service is exposed to the other modules through the JMS communication protocol (back-end to back-end communication).

The requests have to be posted to the UVMSMdrEvent queue. The body of the request has to be of type MdrGetCodeListRequest (Get this class from mdr-model).

As we can see from the JAXb object we have to set the following parameters:

1. acronym : String : the code lists acronym we want to fetch (Mandatory).
2. filter : String : the filter we want to apply if we want to have only specific results (Optional).
3. columnsToFilter : List of Strings : the filter will be applied only to these columns (Optional, defaults to code column)
4. wantedNumberOfResults : Integer : The maximus number of results that is wanted to be returned

 MdrGetCodeListRequest extends MdrModuleRequest implements … {

    @XmlElement(
        required = true
    )
    protected String acronym;

    @XmlElement(
        required = true,
        defaultValue = "*"
    )
    protected String filter;

    @XmlElement(
        name = "columnsToFilter"
    )
    protected List<String> columnsToFilters;

    @XmlSchemaType(
        name = "nonNegativeInteger"
    )
    protected BigInteger wantedNumberOfResults;

The response will be of type MdrGetCodeListResponse : 

  public class MdrGetCodeListResponse extends MdrModuleRequest implements …{

    @XmlElement(required = true)
    protected String acronym;

    @XmlElement(name = "dataSet")
    protected List<ObjectRepresentation> dataSets;

    @XmlElement(required = true)
    protected ValidationResult validation;
                    ….

The data returned will be in the dataSets field. To be noted that each element of the List<ObjectRepresentation>   is one row of the related requested code lists. Each is practically a collection of columns.

Notes : Before sending the message to the UVMSMdrEvent queue you have to set the JMSReplyTo (which sets the queue that will be used to post the response to) property so that MDR can use this queue to send the response to.

MDR Status table

Mdr status table is the table that stores the “meta-data” related to the code lists.

The following information is stored :

  last_attempt,  last_status,  last_success,  object_acronym,  object_description, object_name, object_source

  schedulable ,reference_uuid , end_date, start_date.

This data is then used by the front-end to display the status of each code lists. Ex. When it was last update, what was the last update status (RUNNING, FAILED, SUCCESS, EMPTY_LIST) ecc.

The schedulable field is particularly important since the automatic scheduler will update only those code lists that are schedulable.

NOTE : There is a properties file called exclusionList.properties (in mdr app module) which defines a list of acronyms which have to be excluded from  the acronym synchronization process.  

This list will be used in those cases in which for some reason one or more code lists should not be updated.

The difference between this and the schedulable column in the status table is that this list cannot be overridden and if an acronym is present in it then it will never be synchronized. Whereas the schedulability can be overridden and or changed if needed.

Note that for the changes to be effective (if you change the list in the properties file) you will need to build and deploy again.

Installation and test data

For Mdr module to work you need to deploy MDR APP, MDR Plugin modules and to create the MDR tables in the mdr schema.

1. Run the command “mvn clean liquibase:update -Ppostgres,exec” under the LIQUIBASE folder of MDR DB If you also want to include the test data while creating the schema then run “mvn clean liquibase:update -Ppostgres,exec,testdata”.
2. Deploy MDR APP. You do not need particular steps for this. If you need to exclude some code lists you have to modify accordingly the properties files to include the acronyms (separated by comma) of those lists that you want to exclude from                        synchronization.

              3. Deploy the MDR Plugin module which will provide a means of communication with FLUX TL.

NOTES : If you update mdr codelist tables (new rows / delete rows) manually or through liquibase is important to notice that those changes won’t be taken into account until you do one of the following :

1. Re-deploy mdr app module

                                OR

              2. Call the Lucene re-indexing service (mdr/rest/index/reindex)

The reason for this is that lucene stores only the data that go through it to be saved/updated.                                     

So if you don’t do one of the previous steps then when you query for the code lists you will get old data.