Changelog

What’s New in Release 2.14

This release contains service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

Fixed: Pre-allocated VMs not being used. Customers reported that jobs using machine executor and setup_remote_docker were continuing to spin up (and terminate) new VMs at runtime instead of the pre-allocated pool.

Fixed: Unable to build in AWS region eu-west-2.

Updated: The title of the Builds page in the CircleCI application was changed to Jobs in several places to better reflect the structure of builds.

Updated: Contexts are now checked for unique naming across your installation of CircleCI instead of just within an organization. You will no longer be able to create duplicate context names within an account. Existing duplicates will be honored by the system, but should be changed to avoid future collisions. Please see the Contexts documentation for further information.

Updated: The test-results services cleans up the /tmp directory on restart and the JVM heap size increased from 256m to 512m.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

Migration Tools: The 1.0 migration center was updated with additional content and helpful tools to migrate projects from CircleCI 1.0 to 2.0.

What’s New in Release 2.13*

• Fixed: Due to a certificate not being imported jobs were unable to submit after a container restart.
• Fixed: Stability issues within contexts service.
• Fixed: vm-service issue where AWS ECS and EBS volumes were left used. Added better idempotent client retries, API eventual consistency handling.
• Added: vm-service EBS Volumes cache evictor. Docker Layer Caching EBS volumes get deleted after 14 days of unused. You can control the inactivity period in Installation Admin Settings page.
• Added: Periodic checks to scale down any unused long running instances (e.g. due to reduction in prescaling configurations, or or due to a bug preventing termination of VMs in timely manner).
• Added: Optimizations to speed up provisioning. Resiliency to network/external changes.
• CircleCI is excited to announce that users on AWS can now forward host (Services VM) and docker metrics via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to AWS CloudWatch. This is is the first step into bringing you greater observability into your installation. To enable and configure please go tohttps://example.com/settings#metrics. In this release the following metrics may be enabled: Docker , Networking , Disk , Memory, and CPU. For more information please see the Monitoring documentation.
• CircleCI is enhancing visibility by capturing usage statistics of CircleCI installations. This aggregate data will be used to better support you and ensure a smooth transition off 1.0. This optional feature is automatically available when you upgrade to the latest version of your installation or once a support bundle is downloaded. Below are the metrics being captured. If you would like access to this data in order to better understand your users please contact a CircleCI account representative. Please reference exhibit C within your terms of contract and our standard license agreement for our complete security and privacy disclosures. For more information please see the Usage Statistics documentation.

Weekly Account Usage

Weekly Job Activity

* this release contains migrations and is a required upgrade

The installable version of CircleCI Server 2.9.0 includes the following improvements and bug fixes:

• Resolution for a potential LDAP injection vulnerability
• Addition of support for the following LDAP fields:
  • Group Membership
  • Group Object Classes

See the Authentication document for the steps.

• Addition of default values for OpenLDAP and Active Directory
• Stability and bug fixes for an issue with test results
• Improvements to ready agent logging for support bundles
• Stability fixes and speed improvements for provisioning remote Docker engine
• Reliability improvements for the permissions service client
• Differentiation between the LDAP login and email to remove email as the assumed login value
• Fix for a bug that deselected all projects when a new user logged in for the first time
• Fix for a bug that was preventing Workflows from working correctly when located behind a proxy
• Bug fixes and improvements for Contexts

The installable version of CircleCI 2.8.0 includes the following fixes and improvements:

• An issue involving pre-allocated remote docker, machine VM creation, and other VM usage on AWS when the on-host VM option is selected has been fixed.

• An issue with the VM provider timing out when waiting for SSH to connect has been fixed. This upgrade supersedes any custom values for the Number of SSH attempts for starting VM setting and adds an equivalent field named Timeout for waiting for SSH to connect to VM(s) that you must set.

The installable version of CircleCI 2.6.0 includes the following fixes and improvements:

• It is now possible to create multiple Contexts in the Organization Settings of the CircleCI app. See the Contexts documentation for details and naming conventions.

The installable version of CircleCI 2.5.0 includes the following fixes and improvements:

• This release fixes a problem with DNS resolution failure on alpine-based containers. This problem was observed in the 2.3 release and was fixed for some service containers in 2.4. The 2.5 update fixes the issue for all service containers.
• This release includes improvements to scaling and garbage collection in the VM Service that should improve performance for installations with heavy use of machine executors or remote Docker.

The installable version of CircleCI 2.4.0 includes the following improvements and changes:

• Audit logging - This feature makes available a downloadable log of system events from the CircleCI Admin dashboard. This is the first release of audit logging; more events and improved data in the logs will be delivered in subsequent releases. Find detailed information about the data fields for each event in the Audit Logging section of the Security Features document.

• LDAP integration - It is possible to enable a new CircleCI installation to authenticate users with OpenLDAP or Active Directory credentials. New installations may be configured to have users authenticate through LDAP, including Active Directory support. Refer to the Authentication document for instructions.

• New build-agent - The new build-agent fixes some cases in which users were unable to log into a running job with SSH.

• VM Service improvements - Autoscaling of the VM Service has been improved, so machine executors should have more consistent performance at scale.

The installable version of CircleCI 2.2.0 includes the following improvements and changes:

• New visualization for Workflows - When you click on a workflow in the CircleCI application, a diagram appears with the relationship of your jobs to one another and the complete sequence of the workflow.
• Updates to the onboarding flow - We now show a new user only projects already building on CircleCI rather than a list of all projects found in source control. This change was made to prevent inadvertent adding of projects to CircleCI during the onboarding process. We also added a confirmation dialog for any user opting to follow more than six projects and made it clearer how to add new projects not yet building on CircleCI.
• Support for object stores other than S3 - CircleCI 2.2.0 now supports the same object store options that are supported in 1.x. This is a prerequisite to future releases that will allow non-AWS installations of CircleCI 2.x.
• Updates to the VM Service; ACTION MAY BE REQUIRED IF YOU ARE USING AWS - The VM service has been updated to use volumes rather than persistent instances. This change allows operators to accommodate Docker Layer Caching with a smaller number of long-running VMs. If you installed using the suggested IAM permissions from prior versions you must update your IAM permissions with this upgrade. Refer to the policy template for the complete set of permissions.
• New Contexts feature; ACTION MAY BE REQUIRED IF YOU ARE USING THE HIGH AVAILABILITY CONFIGURATION - The Contexts feature uses a HashiCorp Vault secure storage mechanism. If your installation is running in HA mode, HashiCorp Vault needs to be externalized similar to the Mongo and Postgres databases. Refer to the Configuring High Availability documentation for instructions.

CircleCI 2.0 is a completely updated CI/CD platform that starts every run with a clean image which is automatically provisioned just-in-time for Linux jobs on the CircleCI application installed in your private cloud. The CircleCI 2.0 platform includes significant performance, stability, and reliability improvements along with first-class Docker support, Workflows (pipelines), Resource Allocation, and Contexts. The Management Console (Replicated) also includes new options for enabling 2.0 Builders and appropriate default settings.

If you have CircleCI installed you may access CircleCI 2.0 features on your current installation with no restrictions under your current agreement and support level. However, you must contact your CircleCI account representative for assistance with upgrading, to obtain migration scripts, and to receive a new license channel.

• CircleCI 2.0.0 is only available on AWS and may be installed with Terraform or manually.
• Teams may create a CircleCI 2.0 .circleci/config.yml file in their repositories to add new 2.0 projects incrementally while continuing to build 1.0 projects which use a circle.yml configuration file.

Documentation for the installable 2.0 application is available at CircleCI 2.0 Documentation for the following topics:

• Administrator’s Overview
• Administration FAQ
• Security Features
• Installing CircleCI 2.0 on Amazon Web Services with Terraform
• Configuring High Availability
• Administrative Variables, Monitoring, and Logging
• Introduction to Nomad Cluster Operation
• Backing up CircleCI Data

This is a security update to prevent logging temporary access tokens in build output. Those access tokens would have allowed a user to do a PUT request to overwrite an existing artifact only if all of the following were true:

1. They already had access to the build.
2. They identified the token in the build output.
3. The were to use the token within 30 minutes after the original artifact was uploaded.

The installable version of CircleCI 2.3.0 includes the following improvements and changes:

• Users can now schedule workflows. The underlying cron-service has been added to the installation, though no specific configuration should be needed. See the documentation on Workflows Orchestration for details.
• Support for using private IP addresses in 1.0 builds is now enabled along with 2.0 builds
• A bug in the VM garbage collector was addressed that should help in situations when VMs were being leaked.
• We have disabled Replicated snapshots. Please see the documentation on backups if you have previously used Replicated snapshots.

We have released patches to our AMIs and other infrastructure to address CVE-2016-8655. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
    apt-cache policy linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-07f09d60”
• ap-northeast-2 = “ami-90588ffe”
• ap-southeast-1 = “ami-4f54f82c”
• ap-southeast-2 = “ami-040c3467”
• eu-central-1 = “ami-3465a35b”
• eu-west-1 = “ami-d70421a4”
• sa-east-1 = “ami-2e6bf242”
• us-east-1 = “ami-e68f89f1”
• us-west-1 = “ami-901c4af0”
• us-west-2 = “ami-0c57fc6c”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

We have released patches to our AMIs and other infrastructure to address CVE-2016-5195. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
    apt-cache policy linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-59389e38”
• ap-northeast-2 = “ami-a2e236cc”
• ap-southeast-1 = “ami-a80fa9cb”
• ap-southeast-2 = “ami-53241930”
• eu-central-1 = “ami-5a59a035”
• eu-west-1 = “ami-e0c78993”
• sa-east-1 = “ami-3832af54”
• us-east-1 = “ami-d0396bc7”
• us-west-1 = “ami-76226916”
• us-west-2 = “ami-938420f3”

NOTE: The following AMIs are not maintained by CircleCI, but contain a patched version of stock ubuntu.

• gov-west-1 = “ami-34df6755”
• cn-north-1 = “ami-92f622ff”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

New Feature: The maximum size of files that you can upload for caching during builds has been fixed to 5G. The size is now bumped 20G by default on CircleCI Enterprise and customers can also override the default value to even larger size.

To override the default value, you need to run the following in a REPL of Service Box.

(settings/set {:max-file-cache-bytes <max-size-by-bytes>})

To unset the overridden value and use default, run the following.

(settings/unset! [:max-file-cache-bytes])

This release fixes a bug in the URL structure used to serve build artifacts.

The artifact URL format was recently changed to handle security concerns related to CircleCI’s hosted offering. The security concerns do not affect CircleCI Enterprise customers, but the change caused issues fetching build artifacts in CircleCI Enterprise installations. This release reverts the CircleCI Enterprise artifact format and resolves the issue.

Please make sure that you are using the latest provision-builder.sh and init-builder-0.2.sh before you upgrade to this release:

https://s3.amazonaws.com/circleci-enterprise/provision-builder.sh

https://s3.amazonaws.com/circleci-enterprise/init-builder-0.2.sh

As part of this release we’re changing the behavior of artifacts to only serve an allowed set of content-types. This means we won’t serve .html files as text/html. This is a security risk on CircleCI Enterprise since artifacts are served on the same domain as the rest of the site – as a result, any user or malicious code used as part of your build can push a specially-crafted artifact and gain control of another user’s account.

If this is an issue, you can override this behavior by setting “Serve artifacts with unsafe content-types” in the admin console. We don’t recommend this, but we’re providing it for backwards compatibility.

This release also includes some changes to container networking. Containers now each use a /24 in the subnet 172.16.1.0/16 by default.

If this conflicts with your private network, or if you were editing lxc-net manually in order to fix a prior conflict, you can now use CIRCLE_CONTAINERS_SUBNET and CIRCLE_CONTAINERS_SUBNET_NETMASK_LENGTH on the builders to configure those.