Domino 4.1

See also the fleetcommand-agent Release Notes.

4.1.10 (May 2020)

Known Issues

  • Workspaces that are launched from the "Files" page of a Domino project do not have a scratch space created unless a dataset is added.

4.1.9 (March 2020)

Changes

  • Added ShortLived.iFrameSecurityEnabled feature flag that removes the sandbox attribute from app iframes when set to false.

  • Domino Jobs that are missing the executable file to run will now move to an errored state instead of hanging until killed.

  • Fixed issue where Workspace syncs could fail when syncing very large numbers of files.

  • Fixed issue where a user with Launcher User permissions on a project would incorrectly see a 403 error when trying to open the project.

  • Fixed some issues with uploading files to projects through the UI that would cause unintended Request entity too large errors.

  • Fixed issue where the Windows CLI installer was using an incorrect link for installing the Java runtime.

  • Fixed issue where no_proxy settings in cluster container runtimes were not being respected by Domino containers.

  • Fixed issue where the hardware tier dropdown in launcher dialogs would sometimes be empty.

Known Issues

  • Model APIs, advanced routing mode does not work with the model tester UI in the model API overview page. It will always return the result for the latest version of the model. Note that this is just a UI bug and the actual model URLs for older versions will work as expected. This issue affects 4.1.0 - 4.1.9

  • Some users on certain deployments are encountering a 500 internal server error when they reached the max session timeout.

  • Restarting a node causes apps in Domino to fail.

4.1.8 (February 2020)

Changes

  • Fixed performance issues with workspace log polling.

  • Fixed issue where resetting passwords for local user accounts would fail following upgrade.

  • Fixed issue where deploying an app would fail when running Domino in Kubernetes v1.16+ due to Kubernetes API changes.

  • Fixed various issues where pressing ENTER would not trigger submission on some forms, dialogs, and modals.

  • Fixed issue where scaling the number of instances of a deployed model, or deploying a new model version, would result in a short period of unavailability. The model will now correctly maintain availability of the existing deployment until the new deployment is fully ready.

  • Fixed issue where some app would fail to publish due to issues with formatting on iframe sandbox attributes.

  • Improved error message when attempting to open files as workspaces where the file extension is mapped to multiple pluggable tools in the default environment.

  • Improved error message when trying to access a private project for which your user is not authorized.

Known Issues

  • Workspace will not launch if a snapshot that is marked for deletion is mounted to another project through a shared dataset.

  • An error occurs when installing the Domino CLI with MacOS Catalina.

  • Calls are consuming large amounts of CPU and causing Domino to become unresponsive.

  • When publishing a new revision of a model, the model returns 502 errors for several minutes.

4.1.7 (February 2020)

Changes

  • Domino executions will no longer immediately enter an errored state with Error retrieving run messages if no frontends are available.

  • Fixed an issue where Domino running on EKS with Calico installed would consistently fail to start the first execution on a newly launched node created by cluster scale-up.

  • Fixed an issue where executions would fail to start if the $DEFAULT_COMPUTE_ENV_IMAGE was not present on the node.

Known Issues

  • Scrolling to view all the results of a job is not working properly.

4.1.6 (January 2020)

Changes

  • Leading and trailing whitespace on central configuration keys and values entered in the administration console will now be automatically trimmed.

4.1.5 (January 2020)

Changes

  • Fixed an issue where the link to the executable file from the details panel of a Job would error due to quotes around the filename in the URL not being removed.

4.1.4 (January 2020)

Breaking Changes

  • All arguments supplied to a job or launcher in the Domino UI will be quoted with single quotes. Previously, it was possible to supply shell environment variables that would be expanded.

Changes

  • Fixed an issue where transient issues with heartbeats from execution pods in a deploying state would lead to the execution moving to an error state, aborting deployment.

  • Fixed an issue where workspaces would hang during startup if there was an error executing requirements.txt.

4.1.3 (December 2019)

Changes

  • Changed the default setting of com.cerebro.domino.dispatcher.internalUrl from the dispatcher external URL to http://dispatcher-nucleus.domino-platform, which resolves via service discovery to the actual internal URL.

  • Improved replicator garbage collection.

  • Fixed issue where attempting to log in from the logout page would just reload the logout page.

4.1.2 (December 2019)

Changes

  • Fixed an issue where Domino could not push to remote Git repositories via PAT credentials during a workspace sync.

  • Fixed an issue where using Bitbucket PAT credentials containing / characters would cause Domino to fail to clone repos from Bitbucket. The PAT is now encoded and decoded like a URI.

  • Fixed an issue where the dataset mounting page would sometimes show the owner and project name on a dataset as unknown.

  • Fixed an issue where datasets scratch spaces would sometimes fail to delete.

  • Removed the ability to set pod memory limits in hardware tier definitions. Memory limits are now automatically set to the same value as the pod memory request.

  • Users with the Support Staff role can now access User Activity Reports in the administration console.

4.1.1 (December 2019)

Changes

  • Fixed issue where if the run container in a Domino execution pod stopped unexpectedly, the executor container could be orphaned and left running, causing the pod to not be deleted.

  • Added a ShortLived.LaunchersAccessible feature flag. When set to false, the launchers page in the UI is disabled.

  • Fixed issue where project names that contained only numbers, such as a project named 123, were not quoted correctly in API payloads and therefore various operations on them would fail, including starting runs.

  • Fixed issue where a newline at the end of a Full Delete Message would cause the full delete operation to fail.

  • Fixed issue where Domino runs with Spark integration enabled were passing the run pod name as the spark.driver.host which was unresolvable by external clusters.

  • Added two central config options for enabling setting Domino application roles and organization membership based on SAML attributes:

    • com.cerebro.domino.authentication.oidc.externalRolesEnabled when true will apply a Domino admin role for users with SAML attribute values containing role names mapped to a domino-system-roles user attribute.

    • com.cerebro.domino.authentication.oidc.externalOrgsEnabled when true will set organization membership for users with SAML attribute values containing organization names mapped to a domino-groups user attribute.

  • Fixed an issue where setting organization membership from SAML attributes was incorrectly enforcing case sensitivity when matching organization names to the user attribute values.

  • Fixed an issue where EBS volumes used as Domino execution persistent volumes were sometimes not deleted correctly during garbage collection, and could become orphaned by Domino.

  • Fixed an issue where users would sometimes see Request entity too large errors when trying to upload files larger than 1MB to a project.

4.1.0 (November 2019)

Domino 4.1 delivers powerful enhancements to enterprise authentication by enabling credential propagation through Domino to other systems that user code interacts with. Domino 4.1 also introduces new options for administrators to manage per-user compute resource consumption, and adds the ability to fully purge data from the Domino file store.

New Features

  • Domino 4.1 introduces a full delete operation that allows Domino system administrators to completely remove all instances of a file from the Domino file system. Performing a full delete on a file finds all instances of the file’s contents across all revisions of all projects, erases those contents wherever they appear, and replaces them with a message indicating that the file was subject to a full delete. This will affect all files that have identical contents to the target file, even if they have different filenames. It will not affect files with the same filename if they have different contents.

    To perform a full delete, open the file in question and click the Full Delete button. Full delete can only be performed on one file at a time.

    full_delete_button.png

    You will be prompted to supply a commit message before confirming the full delete. The full delete will remove all instances of the file contents from the Domino file system. Note that this operation does not alter the revision history of any of the affected projects. All commits that contained the file contents will continue to exist, but when viewing the file contents within you will only see the full delete message.

    full_delete_message.png

    One limitation of full delete is that if any Model APIs have been published from project revisions that contained a file that has been fully deleted, the Model API images will continue to contain that file. There is currently no way to permanently purge that image from Domino. Contact support@dominodatalab.com if you have questions about such files.

  • Domino now supports mapping assertions from your SAML identity provider or SSO service to Domino admin roles.

    To automatically assign admin roles to a user, you will need to assert one of the following role values in a SAML attribute statement for that user: SysAdmin Librarian ReadOnlySupportStaff SupportStaff ** ProjectManager

    + Consider this example assertion that could be used to assign a user system admin roles:

    +

    <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> <saml2:Attribute Name="DominoSystemRoles"> <saml2:AttributeValue xsi:type="xs:string"> SysAdmin </saml2:AttributeValue> </saml2:Attribute> </saml2:AttributeStatement>
    <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">
  <saml2:Attribute Name="DominoSystemRoles">
    <saml2:AttributeValue xsi:type="xs:string">
        SysAdmin
    </saml2:AttributeValue>
  </saml2:Attribute>
</saml2:AttributeStatement>

    + Additional instructions on how to finalize these configurations with mappers and enable this feature in the Domino will be available in the Keycloak authentication section of the Domino Administrator’s Guide.

  • Domino now supports mapping assertions from your SAML identity provider or SSO service to Domino organization membership.

    To automatically grant a user membership in a Domino organization, you will need to pass the names of the organizations as values in a SAML attribute statement for that user, such as in the following example.

    <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> <saml2:Attribute Name="DominoOrganizations"> <saml2:AttributeValue>nyc-data-scientists</saml2:AttributeValue> <saml2:AttributeValue>all-data-scientists</saml2:AttributeValue> <saml2:AttributeValue>sensitive-claims-users</saml2:AttributeValue> </saml2:Attribute> </saml2:AttributeStatement>
    <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">
  <saml2:Attribute Name="DominoOrganizations">
    <saml2:AttributeValue>nyc-data-scientists</saml2:AttributeValue>
    <saml2:AttributeValue>all-data-scientists</saml2:AttributeValue>
    <saml2:AttributeValue>sensitive-claims-users</saml2:AttributeValue>
  </saml2:Attribute>
</saml2:AttributeStatement>

    Additional instructions on how to finalize these configurations with mappers and enable this feature in the Domino are available in the Keycloak authentication section of the Domino Administrator’s Guide.

  • Domino 4.1 introduces the ability to propagate AWS temporary credentials for accessing external systems into Domino run environments. This requires federation from AWS IAM to your SAML identity provider.

    To enable credential propagation for a user, you must pass SAML attributes for that user that define the IAM roles they can assume and configurations for temporary credentials. Domino will then enable automatic requests for temporary credentials when the user signs in, and will load them into an AWS credentials file in the user’s run environments.

    Additional instructions on how to finalize these configurations with mappers and enable this feature in the Domino are available in the Keycloak authentication section of the Domino Administrator’s Guide.

  • In Domino 4.1 admins can now optionally configure a maximum number of simultaneous executions per user. This can be used to prevent individual users from monopolizing available compute resources. This feature is enabled by default, but can be turned off with the ShortLived.UserExecutionsQuotaEnabled.

    The maximum number of per-user executions is configurable with a new com.cerebro.domino.computegrid.userExecutionsQuota.maximumExecutionsPerUser option in the central config, and it defaults to 25. If a user attempts to start a new execution when already at the limit, the new execution will be queued in a WaitingForResources state until one of their existing executions finishes.

    over_quota_message.png

    Executions that remain in a WaitingForResources state for more than 24 hours will be terminated. This timeout is configurable with a new com.cerebro.domino.computegrid.timeouts.sagaStateTimeouts.waitingForResourcesStateTimeoutSeconds option in the central config.

    Note that Model APIs and Web Apps are exempt from these limits, as they are considered long-lived services hosted by Domino and are not treated as variable user demand. Jobs, Scheduled Jobs, and Workspace sessions are subject to the limits.

  • A new option is available in the Executions tab of the admin interface to download a Support Bundle. This option downloads a .zip archive of all logs related to the selected execution, including Domino events logs, Kubernetes deployment logs, and Domino application logs.

    support_bundle_link.png

  • Hardware tiers now have an Overprovisioning Pods option. This number represents a number of empty pods that will be deployed to the cluster using the hardware tier’s resourcing requests and node pool settings. If overprovisioning pods are present when a new user-launched execution pod is sent to the cluster for assignment, the user’s pod replaces one of the overprovisioning pods.

    Admins can use this feature to build in some automatically reserved capacity for the hardware tier, so that user executions on the hardware tier always have a node with reserved resources ready to go, and users will never have to wait for autoscaling to spin up a new node.

    overprovisioning_option.png

    Additionally, admins can check the Enable Overprovisioning Pods On a Schedule option to limit deployment of overprovisioning pods to a specified range of hours on specified days of the week. When outside the scheduled period, overprovisioning pods will be removed to allow the cluster to scale down. When the scheduled period begins again, overprovisioning pods will be deployed.

    A new com.cerebro.computegrid.periodicProcessing.overprovisioningInterval option has been added to the central config that sets how often Domino checks to see if new overprovisioning pods are needed. This defaults to 600 seconds.

  • Deployment logs for Domino executions are now available in .csv format. Click the (CSV) link to download the new format. Clicking the Deployment logs link will still download the logs in the original JSON format.

    csv_deployment_logs.png

  • New API endpoint have been added to enable exporting Docker images of Model APIs built in Domino to external registries. Additional details will be available in the Domino API docs.

  • Added a new com.cerebro.domino.modelmanager.requestBufferSize option in the central config that controls the maximum size in bytes of a request block to a Domino Model API. This defaults to 8192 and can be set to a value from 1-65535.

Changes

  • Improved messages on the Workspace startup page that provide more granular information about the startup process.

    improved_workspace_interstitial.png

  • The Cores limit and Memory limit options have been removed from Domino hardware tiers. Domino now automatically sets the limit values equal to the request values to ensure stable and predictable behavior from Domino execution pods and keep assignment simple and reliable.

  • Fixed an issue where having a large number of published models failing to deploy could cause future attempts to publish new models to fail in deployment scheduling. Published models that fail to deploy and end up in a crash loop will now be automatically unscheduled to improve reliability of scheduling newly published models.

  • Domino 4.1 improves performance, UX responsiveness, and user feedback design in many components across the application.

Known Issues

  • The error messaging when there is an environment build failure due to Dockerfile parsing errors is not clear.

  • The links on the Model APIs versions page go to the latest version of the project files instead of the project files for the model version listed.

  • The Support button that is used to contact Domino Technical Support is missing.

  • Multiple hardware tiers can be designated as the "default" hardware tier for a Domino deployment.

  • Assets portfolio becomes unresponsive if the portfolio contains a large number of assets.

  • When restarting a Workspace through the Update Settings modal, External Data Volumes are not mounted in the new Workspace. Follow the steps to mount External Data Volumes. This issue is fixed in Domino 5.9.0.