FossID Documentation

FossID-DA Getting Started

System requirements

FossID Dependency Analysis (FDA) will no longer support CentOS 7 (glibc-2.17) starting from version 1.0.6:

  • FDA-1.0.5 and lower: CentOS 7 with glibc-2.17.
  • FDA-1.0.6 and above: Linux OS with glibc-2.28 or lower.

How to set it up ?

As of Workbench 24.3, FDA (fossid-da) is enabled by default, and no additional configuration is required if you want to use it. If you are using a version of Workbench that does not have FDA installed, follow these steps to install it. Copy fossid-da binary in /fossid/bin folder.

sudo cp fossid-da /fossid/bin

Set permission for fossid-da:

sudo chmod 755 /fossid/bin/fossid-da
sudo chown www-data:www-data /fossid/bin/fossid-da

Copy config/fossid.conf.dist contents (not the whole file) in /fossid/etc/fossid.conf, either via Workbench itself in Edit FossID Configuration or by editing /fossid/etc/fossid.conf.

How to switch between FDA (fossid-da) and ORT?

As of Workbench 24.3, FDA (fossid-da) is enabled by default, and no additional configuration is required if you want to use it.

  1. If you want to use ORT instead of FDA then, either via Workbench itself in Edit FossID Configuration or by editing /fossid/etc/fossid.conf, set:

     webapp_dependency_analysis_tool="sudo -u ort /fossid/lib/ort/bin/ort"

  2. If you want to use FDA instead of ORT then, either via Workbench itself in Edit FossID Configuration or by editing /fossid/etc/fossid.conf, comment or remove:

     ;webapp_dependency_analysis_tool="sudo -u ort /fossid/lib/ort/bin/ort"

  3. If FDA (fossid-da) does not start then, either via Workbench itself in Edit FossID Configuration or by editing /fossid/etc/fossid.conf, set:

     webapp_dependency_analysis_tool="/fossid/bin/fossid-da"

Proxy settings

To activate proxy usage for fossid-da da_use_proxy has to be enabled in fossid.conf:

da_use_proxy=1

Proxy settings can be activated in fossid.conf in two ways:

  1. By using the existing proxy settings that Workbench uses:

     ;Proxy settings
 ;--------------
 ;Specify here your proxy credentials
 ;If you prefer not to type your proxy password here, you could
 ;encrypt your password with using the --password option (see help) 
    
 ;Your proxy hostname or IP
 ;cli_proxy_host=""
    
 ;Your proxy port (i.e. 8080)
 ;cli_proxy_port=""
    
 ;Your proxy username
 ;cli_proxy_user=""
    
 ;Your proxy password (plain text), or
 ;cli_proxy_pass=""
    
 ;Your SSL proxy client certificate
 ;cli_proxy_cert=""
    
 ;Your private key for SSL and TLS proxy client certificate
 ;cli_proxy_key=""

  2. By using fossid-da’s proxy settings in fossid.conf:

     ; Dependency Analysis Proxy Settings
 ; Activate proxy usage
 da_use_proxy=1
    
 ;Your proxy hostname or IP
 da_proxy_host=""
    
 ;Your proxy port (i.e. 8080)
 da_proxy_port=""
    
 ;Your proxy username
 ;da_proxy_user=""
    
 ;Your proxy password (plain text), or
 ;da_proxy_pass=""
    
 ;Your SSL proxy client certificate
 ;da_proxy_cert=""
    
 ;Your private key for SSL and TLS proxy client certificate
 ;da_proxy_key=""
    
 ;Folder containing server cetificates
 ;da_proxy_cert_folder="/path/to/cetificates/"

NOTE: If both Workbench and fossid-da proxy settings are activated, fossid-da settings will be used.

Setting up fossid-shinobi.service

1. If you are using Workbench 24.3 or higher, no additional configuration is required.
2. If you are using Workbench 24.1.x or 24.2.x then fossid-shinobi.service needs to be running on the machine that runs Workbench:
sudo systemctl status fossid-shinobi

If the status shows as active (running), no setup is required.

Otherwise, start the service with the following command.

sudo systemctl enable --now fossid-shinobi.service

Check the status and if it shows active (running), the setup is complete.

sudo systemctl status fossid-shinobi
3. If you are using a Workbench version lower than 24.1:

Run the following install instructions:

sudo install -m 644 fossid-shinobi.default /etc/default/fossid-shinobi
sudo install -m 644 fossid-shinobi.service /lib/systemd/system
sudo systemctl daemon-reload
sudo systemctl enable --now fossid-shinobi.service

NOTE: If fossid-shinobi.service is active and running, the instructions above are not needed.

What does fossid-da currently support?

  • Resolving Javascript dependencies:
    • NPM
    • Bower
    • PNPM
    • Yarn/Yarn2
  • Resolving Java/Kotlin dependencies
    • Maven
    • Gradle
    • Ivy
    • Kotlin
  • Resolving Python dependencies:
    • PIP
    • Pipenv
    • Poetry (only lock files)
    • Hatch (specific envs set in the config)

    • Import detection:

      • NOTE: This functionality is triggered if not valid dependency manifest is detected for the project.

      • NOTE: To activate this functionality add this option in fossid.conf:

        da_py_import_search=1

        Detected imports will be processed and checked if they match any known Python components:

        import imgviz    ===> imgviz - N/A
import torch     ===> torch - N/A
import trimesh   ===> trimesh - N/A
import math      ===> X
import os        ===> X
        • Every detected component from an import will have N/A version since it is difficult the detect the actual version just from the import statement.

  • Resolving Rust/Cargo dependencies

  • Resolving Dart/Flutter dependencies

  • Resolving PHP dependencies:
    • Composer
  • Resolving Ruby/Gem dependencies:
    • Bundler
  • Resolving Yocto dependencies: Limited to dependency extraction from:
    • license.manifest manifest
  • Resolving Soong dependencies: Limited to dependency extraction from:
    • module-info.json manifest
    • Android.bp manifest
  • Resolving Go dependencies (only go.mod, go.sum manifests and lock files):

    • Godep and Gomod

      • If a go.sum manifest is also detected there are two ways of processing the dependencies:

        1. if da_ignore_lock_manifests=1 then the go.sum lock file will be ignored and the go.mod will be processed. Only the first level of transitive dependencies is currently supported.

        2. If da_ignore_lock_manifests=0 , them go.sum will be processed as a lock file and go.mod manifest will be skipped.

        3. If no lock file is detected, then fossid-da will search all relevant imports statements from: .go files

    • Import detection:

      NOTE: To activate this functionality add this option in fossid.conf:

         da_go_import_search=1

      Detected imports will be processed and checked if they match any valid Go components:

         "github.com/sirupsen/logrus"              ===> sirupsen/logrus - N/A
   "github.com/TF2Stadium/Helen/config"      ===> TF2Stadium/Helen - N/A
   "github.com/TF2Stadium/Helen/controllers" ===> TF2Stadium/Helen - N/A
   "github.com/evalphobia/logrus_sentry"     ===> evalphobia/logrus_sentry - N/A
   "github.com/rs/cors"                      ===> rs/cors - N/A
   "os"                                      ===> X
   "os/signal"                               ===> X
   "syscall"                                 ===> X
      • Every detected component from an import will have N/A version since it is difficult to detect the actual version just from the import statement.

    • NOTE: It is recommended that you use a github token to handle Go dependency resolving. A token with only Access public repositories (public_repo) should be sufficient. This must be set in fossid.conf

      ...
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  ; Dependency analysis tool setting                                            ;
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    
  [DEPENDENCY_ANALYSIS]
  ...
    
  ; FossID-DA Git Settings
  da_git_user=""
  da_git_token=""
    
  ...
    • If no token is available, the rate limit will be 60 queries/hour which will heavily slow down the scan.

    • Information about rate limit will also be present in /fossid/logs/fossid-da/debug.log:

      GithubQueryHandler: Limit for github API reached! 60 seconds wait time! | Queries/minute: 1 | Consider adding a github token for more queries and faster processing.

  • Resolving Carthage/CocoaPod dependencies

    • It is recommended that you use a github token to handle Cocoapod (Podfile) dependency resolving. A token with only Access public repositories (public_repo) should be sufficient. This must be set in fossid.conf

      ...
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Dependency analysis tool setting                                            ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
    
[DEPENDENCY_ANALYSIS]
...
    
; FossID-DA Git Settings
da_git_user=""
da_git_token=""
    
...
    • If no token is available, the rate limit will be 60 queries/hour which will heavily slow down the scan.

    • Information about rate limit will also be present in /fossid/logs/fossid-da/debug.log:

        GithubQueryHandler: Limit for github API reached! 60 seconds wait time! | Queries/minute: 1 | Consider adding a github token for more queries and faster processing.
  • Resolving C/C++ dependencies
    • If a lock file is detected, then the dependencies from the lock file will be processed:​
      • conanfile.txt
      • conan.lock
      • vcpkgs.json

    • Detected dependencies from lock files will have valid versions if the version if available in the file.

    • If no lock file is detected, then fossid-da will search all relevant imports statements from:
      • .cpp files
      • .hpp files
      • .h files
      • .hh files

      NOTE: To activate this functionality add this option in fossid.conf:

    •   da_cpp_import_search=1

    Detected imports will be processed and checked if they match any known C/C++ components:

      #include "boost/goo.h"   ===> boost - N/A
  #include "fmt/goo.h"     ===> fmt - N/A
  #include <algorithm>     ===> X
  #include <functional>    ===> X

    • Every detected component from an import will have N/A version since it is difficult to detect the actual version just from the import statement.

      NOTE : For these dependency types:

    • If a lock file is detected it will use the information from it to try and generate the dependency tree based on the manifest. This functionality can be triggered on or off by using this option in fossid.conf:

        da_ignore_lock_manifests=1
    • If no lock file is detected it will generate the dependencies based on the manifest.
    • Filtering for single dependency versions is not currently supported. (For some dependency types, only one version of a dependency is needed and not multiple versions)
    • Dependency graph depth is currently limited.
  • Resolving Elixir dependencies

    • Elixir dependencies are gathered only when the lock file, mix.lock, is available in the project:

      mix.exs 
mix.lock

      This includes transitive dependencies that are found in the lock file(mix.lock)

  • Resolving Haskell dependencies

    • Haskell dependencies are gathered only when the lock file, stack.yaml.lock, is available in the project:

      stack.yaml
stack.yaml.lock

      This includes transitive dependencies that are found in the lock file(stack.yaml.lock)

  • Resolving DotNET transitive dependencies.
    • FDA supports getting direct dependencies and transitive found in the following DotNET manifests:
      • packages.config
      • packages.lock.json
      • project.assets.json
      • Directory.Packages.props
    • FDA supports resolving dependencies from:
      • .csproj
      • .vbproj

    • FDA supports selecting the DotNet framework and standard when resolving dependencies:

      ;FossID-DA .NET Options
 da_dotnet_framework=".NETFramework4.5"
 da_dotnet_standard=".NETStandard1.3"

      NOTE: For a more detailed list of supported files for each ecosystem, please refer to FossID Filtering Dependency Types section.

What fossid-da does not currently support ?

  • Dynamically resolving Elixir dependencies based solely on the mix.exs manifest without relying on the mix.lock file
  • Dynamically resolving Haskell dependencies based solely on the stack.yaml manifest without relying on the stack.yaml.lock file
  • Resolving Go transitive dependencies (that do not have go.mod, go.sum manifests or lock files).
  • Filtering for single dependency versions is not currently supported. (For some dependency types, only one version of a dependency is needed and not multiple versions)

NOTE For dependency types for which FDA does not support dynamically resolving only the manifests, if a lock file is detected (ex: mix.lock, stack.yaml.lock) it will try to generate a flat list of dependencies, but will not generate the dependency graph.

How does it resolve dependencies:

  • fossid-da does not use any package managers. It uses various API/HTTP/HTTPS requests that get information about the component and with that info it resolves the version and transitive dependencies. These requests depend on the type of dependency.
  • Direct dependencies are gathered from local manifests (Ex: package.json, pom.xml, build.gradle, etc. )
  • Transitive dependencies are gathered based on direct dependencies from manifest declarations.

How does it handle manifests and lock files:

To select which dependency type to process please see: FossID-DA-Filtering-Dependency-Types section.

Check current status for filtered package manager types:

You can check the status of this option by going to:

System Information → Initiate:

In the section 4.Verify dependency analysis requirements. you can find:

    Current packages to be processed:
    webapp_dependency_analysis_supported_package_managers='Bundler,DotNet,Gradle,Maven,NPM,PIP,Yarn'

NOTE: If the project has only a lock file and does not have a corresponding manifest file, it might not get processed, depending on the dependency type. It is recommended that both files to be present in the project for better results.

Manifest to lock file map

Bellow is a table that maps what manifests and what lock files are needed:

Dependency type File type File name
Pub Manifest pubspec.yaml
  LockFile pubspec.lock
Go Manifest Gopkg.toml
  LockFile Gopkg.lock
Stack Manifest stack.yaml
  LockFile stack.yaml.lock
Bower Manifest bower.json
  LockFile package-lock.json
NPM Manifest package.json
  LockFile package-lock.json
PNPM Manifest package.json
  LockFile pnpm-lock.yaml
Yarn Manifest package.json
  LockFile yarn.lock
CocoaPods Manifest Podfile
  LockFile Podfile.lock
Composer Manifest composer.json
  LockFile composer.lock
Pipenv Manifest Pipfile
  LockFile Pipfile.lock
Poetry Manifest pyproject.toml
  LockFile poetry.lock
Bundler Manifest Gemfile
  LockFile Gemfile.lock
Cargo Manifest Cargo.toml
  LockFile Cargo.lock
Elixir Manifest mix.exs
  LockFile mix.lock

Choosing/Filtering dependency types

FossID-DA supports choosing or filtering of dependency types.

This can be used by simply uncommenting or adding this line either in Workbench directly or by editing fossid.conf:

webapp_dependency_analysis_supported_package_managers='Bundler,DotNet,Gradle,Maven,NPM,PIP,Yarn'

where:

  • Bundler,DotNet,Gradle,Maven,NPM,PIP,Yarn - represents the types of dependencies that will be processed:

  • Bundler
  • DotNet
  • Gradle
  • Maven
  • NPM
  • PIP
  • Yarn

Example: If the following dependency types are selected:

webapp_dependency_analysis_supported_package_managers='Pub,GoMod'

only these manifests will be considered as valid, and will be processed:

  • pubspec.yaml
  • pubspec.lock
  • go.mod
  • go.sum

A more detailed list can be found in the FossID-DA-Filtering-Dependency-Types section.

Choosing unmanaged dependencies

You can choose to process only unmanaged dependencies by using the following fossid.conf option:

webapp_dependency_analysis_supported_package_managers='UNMANAGED'

This will ignore all other dependency types and focus only on:

  1. Getting import dependencies from the following file types:
  • .cpp
  • .h
  • .hh
  • .hpp
  • .py
  • .pyi
  • .go
  1. Getting dependencies from CMake projects.

How to choose dependency graph depth:

FossID-DA Graph Depth

To change the graph depth for any of the supported dependency types, add one of the following options to fossid.conf either via Workbench or CLI.

Maven:

da_gd_maven=5

PyPI:

da_gd_pypi=3

Ruby:

da_gd_gem=4

Cargo:

da_gd_cargo=4

CocoaPods:

da_gd_cocoapod=4

NPM:

da_gd_npm=7

General:

da_gd_general=4

If the value for any graph depth (gd) options is set to 0, then it will only process direct dependencies and it will skip transitive dependencies.

How to handle dependency scopes:

What are dependency scopes?

Dependency scopes define the context or phase in which a dependency is needed or used within a software project. They help manage when and where dependencies are required, ensuring that only the necessary dependencies are included during different stages of development and deployment. Each tool or package manager (such as Maven, NPM, or Composer) has its way of managing dependency scopes, but the basic idea is the same.

Some examples of how dependency scopes are used by some package managers:

Maven (Java):

Maven uses a dependency scope to control when a dependency is available during different phases of the project lifecycle.

The most common scopes are:

  1. compile (default): The dependency is available in all phases (compiling, testing, and running).

  2. provided: The dependency is needed for compiling and testing but not for runtime (e.g., when the server provides it).

  3. runtime: The dependency is required for running the application, but not during compilation.

  4. test: The dependency is only available during testing.

  5. system: Like “provided,” but you must explicitly provide the JAR file.

  6. import: Used in dependency management for importing external POMs.

NPM (JavaScript/Node.js):

In NPM, dependencies are classified primarily into two categories in package.json:

  1. dependencies: Packages required for your application to run in production.
  2. devDependencies: Packages needed only for development (e.g., testing frameworks, build tools). These won’t be included when deploying the production version.
  3. peerDependencies: Peer dependencies are used to specify that a package depends on a specific version (or versions) of another package.

PyPI (Python):

Python’s packaging system (PyPI) doesn’t have explicit scopes like Maven or NPM, but you typically manage dependencies using:

  1. install_requires: Dependencies required for your package to run, listed in setup.py or pyproject.toml.
  2. extras_require: Optional dependencies for additional features (e.g., dev or test dependencies).
  3. requirements-dev.txt: Often used to list development dependencies like linters or testing tools, while requirements.txt is used for production dependencies.

Rust (Cargo):

In Rust, dependencies are managed via Cargo in the Cargo.toml file, with no formal scopes, but the equivalent concepts are:

  1. dependencies: Required dependencies for building and running your project.
  2. dev-dependencies: Dependencies needed only for testing or development (e.g., testing frameworks).
  3. build-dependencies: Dependencies required for build scripts (code that runs before the main build).
  4. optional dependencies: Marked with optional = true, allowing the user to decide whether or not to include them.

Composer (PHP):

  1. require: Dependencies required for your project to run in production. These are essential packages that your project needs to function.
  2. require-dev: Dependencies needed only for development (e.g., testing frameworks, linters, debugging tools). Not included in production environments.

NOTE: These explanations of dependency scopes are simplified for clarity and may not cover all aspects or use cases. Additionally, not all supported package managers are included in these examples. For comprehensive details, please refer to the official documentation of each package manager.

FossID-DA Dependency Scopes

  1. Process test scopes. This scope is applicable for multiple project types.

     da_ds_test_dependencies=0

  2. Process development scopes. This scope is applicable for multiple project types. shell da_ds_dev_dependencies=0 shell

  3. Process peer dependencies scopes. This scope is applicable for NPM projects.

     da_ds_peer_dependencies=0

  4. Process plugin dependencies scopes. This scope is applicable for Maven projects.

     da_ds_plugin_dependencies=0

  5. Process extensions dependencies scopes. This scope is applicable for Maven projects.

     da_ds_extensions_dependencies=0

  6. Process provided dependencies scopes. This scope is applicable for Maven projects.

     da_ds_provided_dependencies=0

  7. Process runtime dependencies scopes. This scope is applicable for Maven projects.

     da_ds_runtime_dependencies=0

  8. Process library dependencies scopes. This scope is applicable for Maven projects.

     da_ds_library_dependencies=0

  9. Process optional dependencies scopes. This scope is applicable for multiple project types.

     da_ds_optional_dependencies=0

  10. Process dependency_overrides dependencies scopes. This scope is applicable for Dart/Pub projects.

    da_ds_dependency_overrides=0

  11. Process framework dependencies scopes. This scope is applicable for Maven/Scala projects.

    da_ds_framework=0

  12. Process classpath dependencies scopes. This scope is applicable for Maven/Scala projects.

    da_ds_classpath_dependencies=0

  13. Process ext dependencies scopes. This scope is applicable for PHP/Composer projects.

    da_ds_ext_dependencies=0

    This will add information about the ext-{package} dependencies from composer.json manifests:

    "ext-curl": "*" ==> php-curl
    
"ext-json": "*" ==> php-json

  14. Process hatch envs. This scope is applicable for Hatch projects.

    Comma separated envs. Ex: “default,lint”

    da_ds_hatch_envs=""

Handling dynamic scopes

What are dynamic scopes ?

You can create a custom dependency scope like MyDependencyScope by defining a custom configuration. Gradle for example allows you to define and configure custom dependency configurations beyond the default ones like implementation, compileOnly, testImplementation, etc.

dependencies {
    myDependencyScope("org.example:example-library:1.0.0")
}

This option allows dynamic scopes to be processed. This applies to Maven/Scala/Gradle/Kotlin projects:

da_allow_dynamic_scopes=0

List of repository URLs used to query for component info:

  • CocoaPods: https://cocoapods.org/

  • CPP: https://conan.io/center

  • Cargo/Crates: https://crates.io/

  • Debian: https://sources.debian.org/; https://salsa.debian.org

  • Fuget: https://www.fuget.org

  • Nuget: https://www.nuget.org

  • Github: https://raw.githubusercontent.com; https://github.com/

  • Go: https://pkg.go.dev/

  • GoogleMaven: https://maven.google.com/

  • GradlePlugins: https://plugins.gradle.org/

  • Haskell: https://hackage.haskell.org

  • Elixir/Hex: https://hex.pm/

  • LibrariesIo: https://libraries.io

  • Maven: https://mvnrepository.com; https://search.maven.org/

  • NPM: https://registry.npmjs.org; https://www.npmjs.com/

  • PHP/Composer: https://repo.packagist.org

  • Pub: https://pub.dev

  • PyPI: https://pypi.org

  • Ruby: https://rubygems.org/

FossID Workbench Help, version 2025.1.0