Changelog

What’s New in Release 2.19.02

Fixes

• In the LDAP login flow we now use an anonymous form to POST LDAP auth state, rather than sending it as a GET parameter. Previously, when a user authenticated using LDAP, their username and password were sent in plaintext as part of a query parameter in a GET request. As requests are over HTTPS, this left usernames and passwords in request logs, etc. This issue is now fixed.

• Optimizely and Zendesk are now removed from Server release images.

• Fixed an issue in which setting CIRCLE_ADMIN_SERVER_HTTP_THREADS or CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS too high would prevent the frontend container from starting.

• Due to changes in the GitHub API we have removed the use of ?client_id=x&client_secret=y for GitHub, and GHE versions 2.17 and later.

• Fixed an issue that was causing intermittent failures to spin up VMs with DLC in use.

• Fixed an issue that was preventing the customization of proxy settings for Docker containers. See the Nomad Client Proxy and Service Configuration Overrides guides for more infomation.

• Fixed a bug that was preventing job steps for non-failing builds being logged when proxy settings were used for the job container.

• Removed legacy TLS versions 1.0 and 1.1, in addition, enabled 1.2 and 1.3 TLS, and specified the following ciphersuites
  • ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
• Fixed a statsd configuration issue that meant some services were not emitting Telegraf metrics.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client
• Classic Load Balancer is no longer available from this version due to the ciphersuite changes listed above. CircleCI no longer accepts requests from Classic Load Balancer, so you should move to Network Load Balancer (NLB) or Application Load Balancer (ALB).

What’s New in Release 2.19.00

Requirements

• Before upgrading to Server v2.19 your Nomad launch configuration must be updated by following this guide. Server v2.19 uses Nomad version 0.9.3, so if you have externalized Nomad, contact your support engineer before upgrading.

New Features

• You can now customize resource classes for your installation to provide developers with CPU/RAM options for the Jobs they configure. For more information see our guide to customizing resource classes in Server v2.19.

• CircleCI Server installations on AWS can now be configured to work on GovCloud. Using AWS GovCloud enables you to meet a variety of regulatory requirements, such as FedRAMP, ITAR, DoD SRG, CJIS, and HIPAA. Read more about how CircleCI helps customers meet security and compliance needs here.

Upgrades and Fixes

• For v2.19 we have updated Nomad clusters from 0.5x to 0.9, reducing the accumulation of dead jobs and improving overall performance. For more information, see the Nomad changelog.

• The image used to run the RabbitMQ server has been updated to fix vulnerabilities.

• The Server-Mongo-Upgrader service has been removed from the Services machine. Server Mongo Upgrader was a migration service written to facilitate the upgrade from Mongo 3.2 to 3.6, which was done as a part of the CircleCI Server v2.15.0 release in October 2018. This means it is not possible to upgrade directly from a Server version pre v2.15.0 to v2.19.0.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.x until further notice. We have known issues around job step logging and docker commands customization that make the current 2.19.x releases incompatible with an HTTP proxy setup of CircleCI Server. This issue will be fixed in a future 2.19.x patch release and the ‘safe’ version will be flagged up clearly for affected customers.

What’s New in Release 2.18.03

Features

Following our release of Windows support on the hosted offering of CircleCI, running VM-based jobs in Windows environments is now supported on CircleCI Server instances running in AWS.

The version of Windows currently supported is Windows Server 2019. Check out the Customizing VM Service Images doc for the details of how to build your own custom Windows Server 2019 image, and then follow the Server examples in the Getting Started with Windows doc for instructions on how to set up a Windows job on your CircleCI Server installation.

Deprecation

• Server-Mongo-Upgrader will be removed in Server 2.19.0. Only effects customers running < 2.15.0.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs for our recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.

What’s New

• CircleCI 2.1 config keys are now available for reusing commands, executors, and parameters to simplify your .circleci/config.yml file. See the Reusing Config document for examples and instructions. Refer to the Configuring CircleCI reference for 2.1 syntax requirements.

• We have released the Xcode 10 image on CircleCI 2.0. Add the following to your .circleci/config.yml file to select that version of Xcode in your jobs:

  macos:
  xcode: "10.0.0"
  

Note: A known issue, for which a fix is in-progress, prevents shipping iOS apps built on the Xcode 10 image to the App Store.

• For Cloud Container macOS plans, there is an improved flow for adding containers on the Plan Overview page. This can be found under Settings > Plan Overview.

What’s New

• The Xcode 9.4.1 image was released on CircleCI 2.0. To select version 9.4.1 for a job, add the following to your .circleci/config.yml file:

        macos:
          xcode: "9.4.1"

• If you are on the Performance Plan, you can now view available credits on the Plan Overview page of your Organization Settings in the CircleCI app.

Updates

• Fixed a known issue with the CircleCI workflows UI due to . period character in the workflows name.

• Fixed the interaction of the Workflows Rerun dropdown menu by restricting to open-source project users who are part of a GitHub organization.

CircleCI 2.0 is a completely updated CI/CD platform that starts every run with a clean image which is automatically provisioned just-in-time for Linux jobs on the CircleCI application installed in your private cloud. The CircleCI 2.0 platform includes significant performance, stability, and reliability improvements along with first-class Docker support, Workflows (pipelines), Resource Allocation, and Contexts. The Management Console (Replicated) also includes new options for enabling 2.0 Builders and appropriate default settings.

If you have CircleCI installed you may access CircleCI 2.0 features on your current installation with no restrictions under your current agreement and support level. However, you must contact your CircleCI account representative for assistance with upgrading, to obtain migration scripts, and to receive a new license channel.

• CircleCI 2.0.0 is only available on AWS and may be installed with Terraform or manually.
• Teams may create a CircleCI 2.0 .circleci/config.yml file in their repositories to add new 2.0 projects incrementally while continuing to build 1.0 projects which use a circle.yml configuration file.

Documentation for the installable 2.0 application is available at CircleCI 2.0 Documentation for the following topics:

• Administrator’s Overview
• Administration FAQ
• Security Features
• Installing CircleCI 2.0 on Amazon Web Services with Terraform
• Configuring High Availability
• Administrative Variables, Monitoring, and Logging
• Introduction to Nomad Cluster Operation
• Backing up CircleCI Data

Today we updated our Xcode 9.0 beta 3 image to Xcode 9.0 beta 5 (build version 9M202q) with the latest simulators.

You can use this image by specifying Xcode 9.0 in your circle.yml:

machine:
  xcode:
    version: "9.0"

This image tag will be automatically upgraded when new betas are released by Apple, until the final release of Xcode 9.

CircleCI 2.0 is a completely updated CI/CD platform that starts every run with a clean image which is automatically provisioned just-in-time for Linux and Android jobs on the hosted CircleCI application.

Configuration moves into the code in 2.0, so every developer can configure jobs directly in their working branch, teams can try new things without the risk of slowing anyone else down, and business leaders have the ability to operate large global teams with minimal overhead. CircleCI 2.0 prevents you from writing clean-up scripts because every run starts in the same state, eliminating the risk of polluting a test database or leaving files in places that cause problems for the next run.

The CircleCI 2.0 platform includes significant performance, stability, and reliability improvements along with the following new features:

• First-class Docker Support: Choose any image to run your job steps, customizable on a per-job basis on a particular Git branch. Speed up your run times with advanced layer caching. Build docker images with full docker CLI support and full support for docker compose. Support for all programming languages and custom environments that offer more predictable output. See Specifying Container Images for instructions.

• Workflows: Orchestrate jobs and steps with great flexibility using a simple set of new keys in your configuration. Share temporary files between jobs with workspaces for fan-in, fan-out, parallel, and sequential runs. Hold a workflow for a manual approval and restart a workflow from a failed job. See Orchestrating Workflows for details.

• Resource Allocation: Configure your CPU and RAM needs on a per-job basis at the branch level, see the resource_class documentation for instructions. Paid accounts may request this feature from their Customer Success Manager, non-paid users may request to get started by sending email to support@circleci.com.

• Insights: Access interactive charts and analyses in seconds. Visualize trends in your build history to identify and pinpoint bottlenecks. Understand all of your builds at a glance. View the builds that fail most, so you can fix the slowest tests to improve efficiency. See the Collecting Test Metadata documentation for information.

• Debugging with SSH and CLI: Perform local job runs, configuration validation and SSH in to builds for access to log files and debugging running processes. See Using the CLI documentation to learn about running local jobs and refer to Debugging Jobs over SSH for SSH instructions.

• Parallelism: Automatic provisioning of containers as they are freed without waiting for other jobs to finish. See the Parallel Job Execution documentation for examples.

• Advanced Caching: Speed up builds by caching files from run to run using keys that are easy to control with granular caching options for cache save and restore points throughout your jobs. Cache any files from run to run using keys you can control, see the Caching Dependencies documentation for strategies and steps.

• Demonstration Applications: See the Language Guides and Demo Repos to learn CircleCI configuration by example for applications written in Go, Javascript, PHP, Python (Django and Flask), Ruby on Rails, Java, Elixir, and Clojure.

• New Documentation: New and restructured documentation including an Overview, Hello World, Sample 2.0 config.yml File, simplified instructions for Migrating from 1.0 to 2.0, plus examples for Configuring Databases and Creating Custom Docker Images. Refer to the Migration FAQ for common questions and known limitations.

Today we updated our Xcode 9.0 beta image to Xcode 9.0 beta 2 with the latest 10.2 simulators. To start using it, add the following to your circle.yml:

  xcode:
    version: "9.0"

This image tag will be upgraded automatically when new betas are released by Apple, until the final release of Xcode 9.

We have released patches to our AMIs and other infrastructure to address CVE-2016-8655. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
    apt-cache policy linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-07f09d60”
• ap-northeast-2 = “ami-90588ffe”
• ap-southeast-1 = “ami-4f54f82c”
• ap-southeast-2 = “ami-040c3467”
• eu-central-1 = “ami-3465a35b”
• eu-west-1 = “ami-d70421a4”
• sa-east-1 = “ami-2e6bf242”
• us-east-1 = “ami-e68f89f1”
• us-west-1 = “ami-901c4af0”
• us-west-2 = “ami-0c57fc6c”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

We have released patches to our AMIs and other infrastructure to address CVE-2016-5195. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
    apt-cache policy linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-59389e38”
• ap-northeast-2 = “ami-a2e236cc”
• ap-southeast-1 = “ami-a80fa9cb”
• ap-southeast-2 = “ami-53241930”
• eu-central-1 = “ami-5a59a035”
• eu-west-1 = “ami-e0c78993”
• sa-east-1 = “ami-3832af54”
• us-east-1 = “ami-d0396bc7”
• us-west-1 = “ami-76226916”
• us-west-2 = “ami-938420f3”

NOTE: The following AMIs are not maintained by CircleCI, but contain a patched version of stock ubuntu.

• gov-west-1 = “ami-34df6755”
• cn-north-1 = “ami-92f622ff”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.