Before Upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s New in Release 4.3.1

CHANGES

• Removes non-configurable settings from values.yaml for Runner admin to reduce complexity
• Adds logic to check for suspended or deleted GitHub users before fetching config with OAuth tokens

KNOWN ISSUES

• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.3 installation, migration, or operations please see our documentation.

Container Runner SSH Rerun (Preview)

Container runner now supports SSH Rerun. Visit our documentation on setup and usage.

For more information on SSH rerun please see the following documentation.

Status badges display in a README a project’s build status.

This functionality now works as expected for projects that integrate with GitLab & GitHub App.

Fixes

• Fixed a bug that caused the Runners page to appear in the side navigation on the Projects page, which is unsupported on server at this time.
• Fixed a bug that caused the Docker Layer Caching (DLC) settings to appear in Project Settings, which is only supported in server 4.3+.
• Bumped Nomad Autoscaler to version 0.3.7 to resolve an upstream regression that broke the default AWS credential chain for Nomad.

Users can now orchestrate their model deployments across multiple environments through the Amazon SageMaker Orb. They can also track and manage their deployments from within the Releases UI.

See our documentation for more information.

Inbound Webhooks are now available in open preview.

With this new feature, you can trigger a pipeline from anywhere that emits a webhook or run a curl command. With this change, we also updated pages such as the Dashboard to accurately reflect that pipelines can now come from outside your VCS.

Only available to users authenticated through our GitHub App integration. See our community forum and an AI use case for more.

CircleCI’s “Linux CUDA” image has been updated to include the Pytorch, Tensorflow, and HuggingFace transformers libraries out-of-the-box.

Users will no longer need to waste time pulling these large libraries into their execution environment when a job is running. This image is currently only available to jobs that use the gen2 GPU resource class.

To use this new image, specify:

linux-cuda-12:default or linux-cuda-12:edge in the image stanza.

Now in preview, three new resource classes that use GPUs:

1. gpu.nvidia.small.gen2: uses 1 NVIDIA Tesla A10G GPU (4 CPU / 16 GB RAM 200 credits/min)
2. gpu.nvidia.small.multi: uses 2 NVIDIA Tesla T4 GPUs (4 CPU / 16 GB RAM 160 credits/min)
3. gpu.nvidia.medium.multi: uses 4 NVIDIA Tesla T4 GPUs (8 CPU / 30 GB RAM / 240 credits/min)

Only available to users on our Scale pricing plan. See our community forum for more.

Customers can now use a screen reader to read the output of individual steps of a job. The toggle is off by default.

Machine runner 3.0 is now Generally Available, providing improvements to usability, setup, and maintenance of machine runners.

Container runners and machine runners are now built using the same framework. Launch agent users will need to upgrade to the new application, but existing container runner installations will not require any additional action.

Faster feature enhancements: We’ll now bring many network resilience feature enhancements to both types of self-hosted runners (machine and container).

Improved usability: With more uniform logs and metrics, CircleCI’s machine runners ensure consistent and reliable performance and a seamless experience for users regardless of their runner setup.

Setup and maintenance: CircleCI’s machine runner 3.0 simplifies the setup and maintenance process by offering convenient packages that allow you to easily integrate with your preferred cloud templating tools.

Linux, macOS, and Windows support is available. See all the details on the runner documentation.

Machine Runner 1.1 variants will be supported through March of 2024, giving you time to plan a migration.

View our documentation to upgrade.

CircleCI is upgrading the Docker version that is used for jobs that use the Docker executor in November, 2023. This change is critical to ensuring the underlying infrastructure that runs your jobs continues to be secure, reliable, and performant.

This upgrade should be effectively invisible to customers, no action is needed. The upgrade is from Docker 20.10.23 to Docker 24.

See our community forum for more details.

Before upgrading

See the CircleCI server 4.3 release notes and upgrade guide for this release.

There is a background database migration job that is run on install/upgrade to server 4.3. Customers upgrading to 4.3 must ensure this job completes. Full details can be found in the Upgrade guide.

When upgrading from server v3.x you will need to install v4.0 and then follow the recommended upgrade path before installing this version.

What’s new in Release 4.3.0

The v4.3 release introduces upgraded execution architecture. Previously, VM service was responsible for creating virtual machines for machine and remote Docker jobs. This has been replaced with machine provisioner. The new architecture has improved efficiency and stability. This upgrade also brings server inline with CircleCI cloud execution, meaning server will benefit from future updates faster. Some specific benefits include:

• A new UI for step output, which is better at handling larger output and colour.
• New systems for processing job output (step data, caches, artifacts etc.) is much more efficient.
• Machine and remote Docker jobs no longer run through Nomad so compute costs are cut down significantly.
• You do not need to run a Nomad cluster if you just want to run machine (and/or remote Docker jobs) on your server instance.
• DLC is now configured to run through S3, which is cheaper than running on SSD volumes, and there is no longer a limit on the amount of DLC jobs that can be run in parallel. DLC stays more consistent with better pruning and is shared between all jobs in a project instead of each cache being different.
• VM service could fall back to another region/zone, but machine provisioner can handle gaps in availability through:
  • Distributing load across availability zones through round-robining
  • Sending a percentage of traffic to a different zone
  • Instance type fallbacks based on deficits and success

New features

• Server execution architecture has been upgraded from VM service to machine provisioner. For further information, see the Installation guide.
• Server now ships with a custom postgreSQL image which now supports pg_crypto, pg_partman, and pg_cron support extensions.
• Added a REPL toggle to turn on a background which deletes workflow data older than 1 year. This is off by default.
• Config policies are now fully supported on server. _compiled_ is now a reserved word in config policies, so that parameterized configuration is fully available to the config policies engine. For further information see the Config policies overview docs.
• The circleci env subst command is now supported. See the Environment variables docs for more information.
• Added support overriding non-CircleCI-managed images.

Changes

• Nomad Autoscaler has been updated to 0.3.7.
• Server no longer deploys the Prometheus chart.
• The artifact paths have changed to improve download performance. The new path will now be <server-domain>/output/job/<job-id>/artifacts/0/home/circleci/artifacts/<artifact-file>

New services

New services introduced with this release

• feature-flags
• runner-admin
• docker-provisioner
• execution-gateway
• machine-provisioner
• output
• step

Services removed with this release

• output-processing
• picard-dispatcher
• test-results
• vm-service

Database migrations

The following databases will run migrations when upgrading to this version

• distributor
• machineprovisioner - Newly Created
• permissions
• runneradmin - Newly Created
• conductor_production

Fixes

• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is unsupported on server at this time.
• If a job is retried due to an infrastructure failure, it should no longer cause a workflow to get stuck
• Fixed a bug that would try and pull a project’s config with a suspended user’s token.

Known issues

• All pipelines must re-run from start due to architectural changes.
• The job concurrency limit per organization is hardcoded to 2500. This limit is configurable from version 4.3.2 onwards.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

The default Docker version for jobs that use the Remote Docker feature has been updated to Docker 24.0.6.

The default Docker version is used if a specific version is not specified in your .circleci/config.yml file or if you are using the current, edge, or previous tags in the image stanza in your .circleci/config.yml file.

The CircleCI web app now shows how much time is saved on each job that uses the “rerun failed tests” feature.

Upon encountering a failed step, users now can solicit guidance at the press of a button. This takes away the need to copy and paste the error message into your favorite search engine to find out what the problem might be and how to fix it.

Admins can opt out of this experiment by navigating to the Advanced section of Organization Settings.

The feature is powered, currently, by OpenAI. The only data sent to OpenAI is the error message in the output from the failed step. Before the message is sent any sensitive information such as passwords or access keys is removed. None of the data provided is being used to train the model.

Let us know what you think in the Discuss thread.

GitHub authorization with CircleCI has changed for new organizations. When you authorize your CircleCI account with GitHub, you may find this will be done through our GitHub App, rather than the GitHub OAuth app. You can see which account type your organization has by heading to the CircleCI web app and inspecting the URL in your browser:

This style of URL indicates you authenticated with the GitHub App:
https://app.circleci.com/pipelines/circleci/UTxCZDiJ9MLGLC8hR1ZDmg

This style of URL indicates you authenticated with the GitHub OAuth app:
https://app.circleci.com/github/<your GitHub username>

For more information on this integration, please review the documentation

As of August 25, 2023 all projects now have an enhanced experience for viewing steps through the CircleCI UI. Benefits include:

• Vastly improved performance and rendering quality
• No cap on log output
• Terminal control codes are now processed in the UI
• URLs are clickable from the step output

More details are available in our Discuss post.

Before Upgrading

See the CircleCI server 4.2 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.2.1 & 4.1.5

Changes

• Introduced the ability to disable the org-level concurrency limit in distributor. For more information, see the docs.

Fixes

• Fixed a bug that caused the /v2/tasks and /v2/tasks/running endpoints on the runner API to return errors.
• Fixed a bug that caused the self-hosted runners page to appear in the side navigation, but this page is unsupported on server at this time.
• Fixed a bug that caused the Insights page to crash if a user tried to view job resource usage, which is is unsupported on server at this time.
• Fixed a bug where the NGINX Deployment replicas were hard-coded to 1 and did not respect the nginx.replicas value in the Helm values.yaml file.

Before Upgrading

See the CircleCI server 4.2 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

From server v4.2, CircleCI server auto-generates some secrets if they are not present in the Kubernetes namespace. If you are upgrading from v4.1.x to 4.2, and you have managed Kubernetes secrets yourself, you will need to follow the steps outlined in the CircleCI server 4.2 upgrade guide to ensure the system knows some secrets already exist.

What’s New in Release 4.2.0

New Features

• Config policies now available as a preview on server. Use config policies to create organization-level policies to impose rules and scopes to govern which configuration elements are required, allowed, not allowed etc. See the Config policies overview for guidance on setting up config policies. To use config policies CLI commands with server v4.2, append all commands shown in the docs with --policy-base-url https://<CIRCLECI-SERVER-DOMAIN>.com
• You can now find your current server version by running: curl http(s)://<CIRCLECI-SERVER-DOMAIN>/version
• The 2xlarge+ resource class (32 vCPUs, 64 GB RAM) is now available for use with the Linux VM execution environment

Changes

• Introduced the ability to disable the org-level concurrency limit in distributor. For more information, see the docs
• By default, Server 4.2 auto-generates some secrets. If you wish to manage these secrets yourself, you should create these secrets (if not done already) and ensure Helm is aware so that auto-generation is skipped. See the CircleCI server 4.2 upgrade guide for steps.
• RabbitMQ version upgraded from 8.6.4 to 11.15.1.

New Services

The following new services are introduced in this release.

• policy-service

Database Migrations

The following databases will run migrations when upgrading to this version

• builds_service
• cron_service_production
• domain
• insights
• permissions
• conductor_production

Fixes

• Patched various security vulnerabilities.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.2 installation, migration, or operations please see our documentation.

What’s New

The “Rerun failed tests” feature is now generally available. Instead of executing all tests in the testing job from a failed workflow (including tests that passed), only failed tests are executed again. Rerun failed tests helps shorten time-to-feedback when a workflow has flaky tests.

Customers who have adopted the feature are seeing a savings of around 8 minutes per job, and 60% reduction in runtime. Learn more about rerun failed tests in our official documentation.

What’s New

The quarterly update for Linux images is now available. Note that Ubuntu 20.04 and 22.04 are both available on amd64 and arm64 architecture under the edge tag, and are scheduled to be promoted to the GA release next week.

More details on this release, including software updates, can be found in the Linux 2023 Q3 release Discuss post.

What’s New

Image updates are available for the June 2023 snapshot of Windows Server 2022, as well as the current tag. This includes Windows security and OS updates, and package updates for Windows Server 2022.

For a full list of updated packages and image usage examples, visit the Discuss thread for the June 2023 image.

What’s New

CircleCI Personal and Project API access tokens have a new format. Previously, token values included a 40-character hex string. Newly created tokens will now include a CCIPAT or CCIPRJ prefix for personal access tokens or project access tokens, respectively:

CCIPAT_<base58-UUID>_<40-char-hex string>

For example: CCIPAT_ULUhR6rJLxYbyzyrP19iMZ_8fb1e3510325f7f09361de22b9420346c53ca2cb

Existing tokens are not impacted.