About the Data Microservice¶

The Data Microservice is a lightweight, standalone service that provides FMR's data validation and data transformation web services without the full metadata registry.

The main use case is where institutions need to validate, convert or map large volumes of data, and want to run that workload separately from the registry. Data processing is CPU and memory intensive. Moving it to one or more microservice instances keeps the registry responsive for metadata maintenance, and the processing tier can scale on its own, for example behind a load balancer or as replicas in Kubernetes.

Note

The Data Microservice processes data only. It does not store data or structures, and it has no user interface.

Data validation
Data format conversion and structure mapping
Structural metadata maintenance or dissemination
Asynchronous (job based) processing

How it works with FMR¶

The Data Microservice is a client of an FMR instance. FMR stays the single source of truth for structural metadata, and the microservice gets the structures it needs from FMR on demand.

flowchart LR
    C[Client] -- "POST data<br>/validate, /transform" --> DMS[Data Microservice]
    DMS -- "Structure queries<br>/sdmx/v2/structure/..." --> FMR[Fusion Metadata Registry]
    DMS -- "Poll for changes<br>/ws/registry/tx/..." --> FMR

Structure retrieval. When a dataset is submitted, the microservice works out which Dataflow, Data Structure Definition or Provision Agreement the data refers to. It then queries FMR's SDMX v2 structure API for that structure and everything it depends on: codelists, concept schemes, constraints and, for transformations, structure maps and representation maps.
Caching. Retrieved structures are cached in memory, so later requests for the same dataflow don't need another round trip to FMR.
Keeping in sync. A background task polls FMR's Transaction Web Service every 30 seconds by default. When a structure is created, modified or deleted in FMR, the microservice loads that transaction and updates its cache. Changes made in FMR therefore show up in validation and transformation results within one polling interval, with no restart needed.
Processing. Validation and transformation use the same engine as FMR. The request headers, accepted formats, validation report and output are identical to FMR's own web services.

Access to structures

The microservice makes anonymous requests to FMR. All structures needed for processing, and the transaction API, must therefore be publicly readable on the FMR instance it is linked to.

Tip

One FMR can serve any number of Data Microservice instances. Each instance keeps its own cache, so the service holds no shared state and can be scaled horizontally.

Deployment¶

The Data Microservice is a Java Spring Boot application with an embedded web server. It needs only a Java 17 or later runtime. No separate application server is required.

It needs network access to a running FMR instance. Use an FMR 12.4.0 or later release, since the microservice relies on the transactions API.

Docker¶

Docker images are published on Docker Hub as sdmxio/microservice-data for linux/amd64 and linux/arm64. Images are tagged with the FMR release version and with latest.

Get Docker Desktop
At the operating system command prompt, run the following, replacing the APP_FMR value with the URL of your FMR instance:
```
docker run --rm -p 8000:8000 --name dms -e APP_FMR=http://host.docker.internal:8080 sdmxio/microservice-data:latest
```
Check that the service is up by browsing http://localhost:8000/actuator/health/readiness

Set APP_FMR when running in a container

The default FMR URL is http://localhost:8080/FMR. Inside a container, localhost refers to the container itself, not the host machine. Always set APP_FMR to a URL the container can reach, for example:

http://host.docker.internal:8080 for an FMR running on the Docker host (Docker Desktop)
http://fmr:8080 for an FMR container called fmr on the same Docker network
https://registry.example.org/FMR for a remote FMR

Include FMR's context path if it has one, for example /FMR or /FusionRegistry.

The image is based on the Red Hat UBI 9 OpenJDK 17 runtime and runs as a non-root user (UID 185). It listens on port 8000.

Environment variable	Purpose
`APP_FMR`	Base URL of the FMR instance to retrieve structures from (required in a container)
`SERVER_PORT`	Overrides the default listening port 8000
`APP_SENDERID`	Sender ID written in the header of SDMX messages the service generates
`APP_TXMONITOR_ENABLED`	Set to `false` to turn off polling FMR for structure changes
`APP_TXMONITOR_INTERVALMS`	Polling interval in milliseconds (default `30000`)
`JAVA_OPTS_APPEND`	Extra JVM options, for example `-Xmx4g` to set the maximum heap size

Logs go to the container's standard output and to /deployments/log/log.out. Mount a volume on /deployments/log if log files need to persist beyond the container's lifetime.

Running alongside FMR¶

The following Docker Compose example runs FMR with a Data Microservice on the same network:

services:
  fmr:
    image: sdmxio/fmr-mysql:latest
    ports:
      - "8080:8080"

  dms:
    image: sdmxio/microservice-data:latest
    depends_on:
      - fmr
    environment:
      APP_FMR: http://fmr:8080
    ports:
      - "8000:8000"

Running the JAR¶

The microservice can also run directly on any machine with Java 17 or later:

java -jar microservice-data.jar --app.fmr=http://localhost:8080/FMR

Any configuration property can be supplied in the same way, as a --property=value command-line argument, as an environment variable, or in an application.yaml file.

The application looks for application.yaml in the following locations, with later entries taking precedence:

The classpath root and the classpath /config package
The current directory
The config/ subdirectory of the current directory
Immediate child directories of the config/ subdirectory

Configuring the Data Microservice¶

Configuration properties¶

The default application.yaml is shown below:

#The port the application will run on
server:
  port: 8000

app:
  #Base URL of the FMR instance to retrieve structures from
  fmr: http://localhost:8080/FMR

  #The sender ID used in the Header of SDMX messages
  senderId: BIS

  #Identify this service in the validation output
  productName: Data MicroService
  productCode: DMS

  #The User-Agent HTTP header sent with requests to FMR
  userAgent: DataMicroService

  #Polls FMR for structure changes to keep the cache up to date
  tx-monitor:
    enabled: true
    interval-ms: 30000

logging:
  file:
    name: log/log.out
    path: log
  logback.rollingpolicy:
    max-file-size: 10MB
    max-history: 4

#Provides a health check web service
management:
  endpoint:
    health:
      probes:
        enabled: true
  endpoints:
    web:
      exposure:
        include: health

Property	Description
`server.port`	Port the application listens on
`app.fmr`	Base URL of the FMR instance to retrieve structures from and keep in sync with
`app.senderId`	Sender ID used in the header of generated SDMX messages
`app.productName`	Product name identifying the service
`app.productCode`	Product code identifying the service
`app.userAgent`	User-Agent HTTP header sent when querying FMR. Useful for identifying microservice traffic in FMR or proxy logs
`app.tx-monitor.enabled`	Turns polling FMR for structure changes on (`true`) or off (`false`)
`app.tx-monitor.interval-ms`	Polling frequency in milliseconds
`logging`	Log file location and rolling policy
`management`	Enables the health check web services

Disabling the transaction monitor

If app.tx-monitor.enabled is false, the microservice doesn't see structure changes made in FMR after it first retrieved them. Turn it off only when structures are known not to change, and restart the service to pick up any changes.

Web services¶

The Data Microservice exposes the following web services. Validation and transformation behave exactly as they do in FMR. Only the entry point differs.

Web service	Data Microservice entry point	Equivalent FMR entry point
Data Validation	`[app url]/validate`	`/ws/public/data/validate`
Data Transformation	`[app url]/transform`	`/ws/public/data/transform`
Health check	`[app url]/actuator/health`	n/a

Both data services accept the same HTTP headers as FMR, including Structure, Data-Format, Accept, Inc-Metrics, Inc-Valid, Inc-Invalid and Zip. Data can be sent in the POST body or attached as a multipart file in the uploadFile field. See the linked pages for full details of headers, supported formats and responses.

Note

These are synchronous services, and the asynchronous validation and transformation web service isn't available on the microservice. For very large datasets, give the service enough memory and set suitable HTTP timeouts on clients and proxies.

Validate data¶

Validate an SDMX-CSV file against a dataflow:

curl -X POST \
   --header "Content-Type: application/text" \
   --header "Structure: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)" \
   --data-binary "@my-data.csv" \
   http://localhost:8000/validate

The response is the JSON validation report described in the Data Validation Web Service documentation.

Transform data¶

Convert an SDMX-CSV file to SDMX-ML 3.0:

curl -X POST \
   --header "Content-Type: application/text" \
   --header "Accept: application/vnd.sdmx.data+xml;version=3.0.0" \
   --header "Structure: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)" \
   --data-binary "@my-data.csv" \
   http://localhost:8000/transform \
   > my-data.xml

If FMR holds a Structure Map from the source to a target structure, the transformation can also map the data to the target structure. The headers that control mapping are described in the Data Transformation Web Service documentation.

See Call Web Services using curl for more tips on using curl.

Health check API¶

The health check web services are designed for container orchestrators such as Kubernetes, and for load balancers.

Liveness

Indicates whether the application is alive and running. If this check fails, the application should typically be restarted.

  https://[app url]/actuator/health/liveness

Readiness

Indicates whether the application is ready to accept traffic and process requests.

  https://[app url]/actuator/health/readiness

Response Example

{
"status": "UP"
}

Possible Status Values

Status	Description	Default HTTP Status
UP	Service is healthy and operational	200 OK
DOWN	Service is unavailable or unhealthy	503
OUT_OF_SERVICE	Service is running but intentionally unavailable	503
UNKNOWN	Health state could not be determined	200 OK

Security considerations¶

The Data Microservice doesn't authenticate its callers. If access needs to be restricted, put it behind an API gateway, reverse proxy or network policy that enforces authentication.