FossID Documentation

Scan Server Installation

Assumptions

  • This example is applicable to:
    • Debian 11
    • Debian 12
    • Ubuntu 20.04
    • Ubuntu 22.04
    • Ubuntu 24.04
    • RedHat Enterprise Linux 8 and 9
    • RedHat Enterprise Linux Server 8 and 9

Introduction

The FossID scan server provides a HTTP interface used to query the FossID knowledge base. The FossID CLI will send HTTP queries to a scan server, which might be routed through a reverse proxy to support running the queries against the standard http/https ports. Below are two alternative setups for a standalone scan server:

Without reverse proxy:

+------------+                     +----------------+
|            |  Wrapper service    |                |
| FossID CLI +-------------------->| FossID wrapper |
|            | Port 8080 (default) |                |
+------------+                     +----------------+

With reverse proxy:
+------------+                    +--------------------+                     +----------------+
|            |  HTTPS port 443    |                    |  Wrapper service    |                |
| FossID CLI +------------------->| Reverse http proxy +-------------------->| FossID wrapper |
|            | (alt HTTP port 80) |                    | Port 8080 (default) |                |
+-------------                    +--------------------+                     +----------------+

If the scan service should be reachable from outside of the local machine, the second alternative using a reverse is strongly recommended.

If the server hosting the scan service also will host a FossID Workbench, combined with the requirement of having the scan service being reachable from the outside, the http server hosting the FossID Workbench can be configured as a reverse proxy. If that is the case, please start by installing the FossID Workbench by following the separate installation instruction for it, then come back to this instruction. In this case, a schematic image of the FossID Workbench running on the same machine as the scan server would (with the Nginx web server set up as a reverse proxy) look like this:

+--------------+
|              |  HTTPS port 443                                                    +------------------+
|  FossID CLI  +-------------------+                             Wrapper service    |                  |
|              | (alt HTTP port 80)|  +---------------------+ +-------------------->|  FossID wrapper  |
+---------------                   |  |                     | | Port 8080 (default) |                  |
                                   +->| Nginx web server    +-+                     +------------------+
                                      | and reverse proxy   |
                                   +->|                     +---+                 +------------------+
 +-------------+                   |  +-----------+---------+   | FastCGI socket  |                  |
 |             |  HTTPS port 443   |              |             +---------------->|  PHP-FPM FastCGI |
 | Web browser +-------------------+              |                               |                  |
 |             | (alt HTTP port 80)               |                               +---------+--------+
 +-------------+                                  |                                         |
                                                  |                                         v
                                                  |                                  +-------------+
                                                  | Files                            |             |
                                                  +--------------------------------->| /fossid/www |
                                                                                     | File system |
                                                                                     |             |
                                                                                     +-------------+

The FossID CLI can be executed manually on the command line to perform scans, or by the FossID Workbench as part of a scan.

Install dependencies

Prerequisites on system wide settings

firewalld

If firewalld is set up, and if a reverse proxy is to be used, open ports for http and https:

sudo firewall-cmd --add-service=http --zone=public --permanent
sudo firewall-cmd --add-service=https --zone=public --permanent
sudo firewall-cmd --reload

If firewalld is set up, and if a reverse proxy is NOT to be used, but the wrapper should be reachable from outside of the server, open the port of the wrapper. If the wrapper port is changed from the default, make sure to replace 8080 in the command below:

sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent
sudo firewall-cmd --reload

SELinux

Please disable SELINUX in /etc/selinux/config and reboot the system. This is needed as FossID do not yet have a SELinux policy in place.

Install required packages

Debian / Ubuntu

sudo apt install default-jre-headless unzip -y

RHEL

sudo yum install java-11-openjdk-headless unzip -y

Install reverse proxy (optional)

The wrapper (see below) will run as an unprivileged process, which means that it will bind to a port number above 1024 (8080 by default). To be able to access the scan service on port 80 (http) and/or 443 (https), a reverse proxy is recommended. Different solutions are available, please see below for a few examples:

nginx

nginx is available on most platforms using apt/yum install nginx. On RHEL, the package is available through epel.

To install on Debian or Ubuntu, run:

sudo apt install nginx -y

To install on RHEL, run:

sudo yum install epel-release -y
sudo yum install nginx -y

The reverse proxy configuration is done as part of a location-block inside a server-block, for example:

    server {
            listen 0.0.0.0:443 ssl;
            ssl_certificate /etc/tls/CERTIFICATE.pem;
            ssl_certificate_key /etc/tls/CERTIFICATE.key;

            # Running over HTTPS is recommended, but if you wish to run plain
            # HTTP, remove the above lines and uncomment this:
            #listen 0.0.0.0:80;

            server_name scanserverhostname.example.org;

            location / {
                    proxy_pass http://127.0.0.1:8080;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_read_timeout 11m;
            }
    }

This configuration can be set in /etc/nginx/nginx.conf on RHEL and /etc/nginx/sites-enabled/default on Debian.

If you have just installed nginx, you will need to enable and start it:

sudo systemctl enable nginx
sudo systemctl start nginx

If nginx is already started, you can instruct it to re-read the configuration by issuing a reload:

sudo systemctl reload nginx

Haproxy

Haproxy is a load balancing reverse proxy available on most platforms using apt/yum install haproxy.

To install on Debian or Ubuntu, run:

sudo apt install haproxy -y

To install on RHEL, run:

sudo yum install haproxy -y

Haproxy is configured mostly by adding frontend and backend definitions in the main configuration file, /etc/haproxy/haproxy.cfg. Below is a simplistic example of two frontends, and a scan server backend:

    frontend https-in
            bind 0.0.0.0:443 ssl crt /etc/tls/COMBINED-CERT-AND-KEY.pem alpn http/1.1
            default_backend scan_servers

    # Running over HTTPS is recommended, but if you wish to run plain HTTP,
    # remove the above https-in frontend definition and uncomment this
    #frontend http-in
    #        bind 0.0.0.0:80
    #        default_backend scan_servers

    backend scan_servers
            balance leastconn
            # Have the maxconn value match the
            # wrapper_max_concurrent_requests_scan value configured below
            server node5 127.0.0.1:8080 maxconn 10 check

Besides configuring the frontends and backend, it is important to increase the timeouts in the defaults section:

    defaults
            timeout connect 5s
            timeout server 11m
            timeout http-request 11m
            timeout client 11m
            timeout queue 11m

If you have just installed haproxy, you will need to enable and start it:

sudo systemctl enable haproxy
sudo systemctl start haproxy

If haproxy is already started, you can instruct it to re-read the configuration by issuing a reload:

sudo systemctl reload haproxy

On some operating systems, haproxy is not configured for reload through systemd. If so the error message “Job for haproxy.service invalid.” will be displayed. As a workaround, restart haproxy instead:

sudo systemctl restart haproxy

Access deliverables

Download the FossID installation package from the access site provided in the delivery mail. Additionally, download the fossid-alfred.key from the same site. You will also need KB volumes that either should have been delivered to you on physical drives, or that you have downloaded from the KB volume download site.

Install FossID deliverable

Debian / Ubuntu

sudo apt install ./fossid-release_{VERSION}-offline_amd64.deb -y

RedHat

sudo useradd -r -s /sbin/nologin www-data
sudo yum localinstall fossid-release-{VERSION}_offline.x86_64.rpm -y
sudo chown -R www-data:www-data /fossid

Add the license file

Copy the fossid-alfred.key to the /fossid/bin directory where the fossid-alfred binary is.

Start services

If the FossID deliverables were installed after the dependencies, some services might not have started. To ensure that everything is started correctly, issue this command on the host:

sudo systemctl start fossid-wrapper.service fossid-shinobi.service

Install volumes

The FossID knowledge base (which is the database queried by the scan server) is divided into “volumes” that contains different parts of the knowledge base. The volumes are delivered separately from the software, but a short guide is included here for completeness.

Unpacking

The volumes are delivered as zip files. It is recommended to unpack them in the /fossid/vols directory. For example:

cd /fossid/vols
sudo unzip /path/to/VOLname.zip

Configuration

For the scan service to be able to use the volumes, they need to be listed in fossid.conf. Locate the row alfred_volumes in /fossid/etc/fossid.conf and set it to a comma-separated list of all volume paths. For example:

alfred_volumes = /fossid/vols/VOLMAIN,/fossid/vols/VOLVSF

List all volumes that you have received, including any security volumes. All volumes must be listed using the alfred_volumes configuration, the old alfred_security_volumes parameter is deprecated and is no longer used. If the instructions for the volumes instruct you to specify them in a certain order, make sure to adhere to that order.

The folders pointed to should contain one or two files and one or more subdirectories of the pattern “?H0” Where “?” is a capital letter. If you are having trouble scanning, make sure this is the case for all listed volumes.

Configure scan server

To set up your scan server, the following configuration options must be set in /fossid/etc/fossid.conf:

  • Alfred
  • Wrapper

If token specific alfred configuration is of interest to allow differentiation in e.g. volume access, the alfred configuration section can be copied to a TOKEN specific configuration file at the location: /fossid/etc/custom/TOKEN.conf See section 6 for more information about token creation guidance.

Configuration content overview

This is an overview of the default locations of scan server content of interest

/fossid
├── bin
│   ├── fossid-cli        (mandatory)
│   ├── fossid-alfred     (mandatory)
│   └── shinobi.jar       (mandatory)
├── etc
│   ├── fossid.conf       (default config if not token specific is present)
│   └── custom            (optional)
│       ├── TOKEN1.conf   (custom config for TOKEN)
│       ├── TOKEN2.conf
│       └── TOKENx.conf
├── tokens                (mandatory)
│   ├── TOKEN1            (TOKEN is a 10 digit hex number, see section 6)
│   ├── TOKEN2
│   └── TOKENx
├── vols                  (recommended directory for FossID KB, see section 5)
├── mdb                   (mandatory, must be writable by www-data user)
└── wrapper               (location of wrapper)
    └── wrapper.jar       (wrapper binary)

Alfred configuration

Alfred is FossID’s scan engine. It is by default located in /fossid/bin/fossid-alfred.

Configuration of it is located under section [Alfred] in either of the following location:

  • Generic configuration; /fossid/etc/fossid.conf
  • Token specific configuration (see section 6 for token creation): /fossid/etc/custom/TOKEN.conf

1) Make sure that volume paths are correctly set. This is applicable to alfred_volumes

2) Make sure that alfred_mdb path is at an available location and that the directory is writable by the www-data user.

3) Enable shinobi by setting the alfred_shinobi_host parameter to the default value of 127.0.0.1:9900.

Wrapper configuration

The Wrapper mainly serves as FossID’s scan server access function but also have a few other purposes:

1) Control client access for to scan server. Scan access control is done by tokens, see section 6 for more information.

2) Provide queuing of different requests, and configure the number of concurrent requests for scan requests, mirror requests and non-scan-nor-mirror requests:

  • wrapper_max_concurrent_requests_scan
  • wrapper_max_concurrent_requests_mirror
  • wrapper_max_concurrent_requests_other

For the server to perform optimally, tuning the concurrent scan request parameter is recommended. A good starting point is half of the server RAM minus 16GB divided by two. For example, 64GB minus 16 divided by two is 24.

3) Log requests to /fossid/logs, details can be configured through /fossid/etc/wrapper_logging.xml using Logback XML syntax. By default, logs will be written to:

  • /fossid/logs/server.log – Generic server log
  • /fossid/logs/TOKENx.log – Token-specific logs
  • /fossid/logs/TOKENx/*.log.gz – Archive of token-specific logs

Changing the listening port used by the wrapper (optional)

If there are other services using the wrapper port, the port number can be changed by editing the file /etc/default/fossid-wrapper and changing the --port argument. Then issue the following commands:

sudo systemctl daemon-reload
sudo systemctl restart fossid-wrapper.service

Changing the listening interface used by the wrapper (optional)

If no reverse proxy is installed and configured (see above), but the wrapper will need to be reached from another machine, the wrapper listening IP can be altered from the default loopback address to any other device in the file /etc/default/fossid-wrapper. To listen on all addresses, use 0.0.0.0. Then issue the following commands:

sudo systemctl daemon-reload
sudo systemctl restart fossid-wrapper.service

Create user tokens

User access to the scan engine is based on the presence of files with a 10 digit hex tokens located in the /fossid/tokens/ directory. These can be generated manually, such as 1234567890, or for a more randomized token ID, run:

dd status=none if=/dev/random bs=1 count=5 | od -tx1 -An | tr -d ' '

Or, if the xxd command is installed:

dd status=none if=/dev/random bs=1 count=5 | xxd -p

The token files contain JSON data, describing the properties of the token. The simplest content is an empty JSON document. For example:

$ cd /fossid/tokens
$ dd status=none if=/dev/random bs=1 count=5 | xxd -p
d127c941fd
$ echo '{}' > d127c941fd

If the scan server is set up with Vulnerability Snippet Finder volumes, the feature needs to be enabled on a per-token basis. To do that, set the “vsf” value of the JSON document to 1. For example:

$ cd /fossid/tokens
$ dd status=none if=/dev/random bs=1 count=5 | xxd -p
42b5737c5e
$ echo '{"vsf": 1}' > 42b5737c5e

If you want to distribute scan capacity or keep whitelisting rules and match database separated between users, separate tokens allow you to do this.

Do note that tokens that do not fulfill the 10 digit hex pattern will be rejected by the CLI and result in no requests being sent.

Verify installation

The host for your scan server looks as follows:

[http://]<ip or hostname>[:port]

If the protocol specification (http://) is left out, https is the default. If port is left out, the default ports for http (80) and https (443) are used.

To verify the installation, you can use the FossID CLI. If you are running locally on the scan server, and you have a token, e.g. 1234567890 in the /fossid/tokens/ directory, it is possible to test the scan server by directly querying the wrapper (if the default port of 8080 was not changed):

/fossid/bin/fossid-cli --host http://127.0.0.1:8080 --token 1234567890 --test-scan

The command will send a pre-generated scan query to the server using hashes obtained from the OpenSSL source code. So the expected result is a number of json messages (one per line) that points to the “openssl” component. In case of an unsuccessful scan, the answer will only be one json line containing "match":"error", with a verbose error in the error_stderr key of the json response.

If a reverse proxy was set up, try using the host name of the machine and leave out the port name:

/fossid/bin/fossid-cli --host scanserverhostname.example.org --token 1234567890 --test-scan

Alternatively, if only http was set up:

/fossid/bin/fossid-cli --host http://scanserverhostname.example.org --token 1234567890 --test-scan

If you have defined the cli_server_host and cli_token in /fossid/etc/fossid.conf, you can run:

/fossid/bin/fossid-cli --test-scan

The VSF volume may also be tested by adding –vsf:

/fossid/bin/fossid-cli --test-scan --vsf

In this case, one or more of the answer json lines should contain a vulnerability key listing vulnerabilities.

Scan performance tuning

Scan server aspects

A typical bottleneck for the scan server is the type of disk used. The used storage configuration has a great impact on the overall scan performance and the number of concurrents scans. We strongly suggest the use of high performance SSD disks.

To run worst-case benchmark, execute the following command on the scan server:

/fossid/bin/fossid-cli --test-benchmark

Note that if the token and host is not configured in fossid.conf, you will need to supply that information using --host and --token, see above.

To tune the number of concurrent requests being serviced, see the scan server configuration chapter above. In particular the information about how to tune the wrapper_max_concurrent_requests_scan parameter.

Client aspects

The client side can also configure how scans are issued, allowing scan capacity distribution on a on a more granular level than per token.

If the client side goes beyond its capacity limit of the scan server, it will receive one of the following responses:

  • Server queue is full
  • Server is too busy

The client side can use these responses reduce the number of scan requests to match the current scan server capacity, which may vary depending on the number of simultaneous clients.

When scanning using CLI, the reduction of scan request can be done by modifying cli_threads in the settings.

When scanning with API and/or Webapp, the results will be available after the scan has completed. For every file that has not been successfully scanned, reasons are available to analyze and a rescan for such files can easily be started.

FossID Workbench Help, version 2025.1.0