Resolved
We believe all workers should now be running as expected.
This issue was caused by an error that was introduced yesterday in our management API at approximately 5:45 PM PDT. As a result of this error, changes that were made to the number of dynos or workers for applications on the Aspen and Bamboo stacks were not acted upon. The previous numbers continued to be in effect.
While our log data indicates that a very small number of applications were affected, this is not an acceptable problem. We are introducing additional testing around this area of our API service and the integration between it and our process management service. We are also adding additional monitoring of the production environment so that we can more quickly detect problems of this nature in the future.
Thank you for your patience.
Issue
We are still working on fully resolving this issue for the small number of applications that were affected. We believe we are in the home stretch.
Issue
The patch is now in production and confirmed working. We have to perform some additional fixes to correct existing worker processes. In the mean time, you can scale your workers down and then back up again to get working immediately.
For example, if you wanted 2 workers you could do the following from the command line with the Heroku client:
$ heroku workers 0
$ heroku workers 2
We will update this incident when all workers have been restarted.
Issue
We have identified the problem and are preparing to deploy a fix to our production environment. We expect to have this patch deployed within approximately 15 minutes. Once patched, scaling operations should begin to work normally again.
We do not currently have an ETA on fixing existing worker processes. Once the patch has been deployed to production, however, you will be able to scale your worker processes manually to correct the problem.
We will post a status update once the patch has been deployed and again when we have corrected all existing workers.
Issue
We are also investigating problems scaling the number of dynos in the Bamboo stack at this time. We will post more information as it becomes available.
Issue
We are currently investigating problems relating to execution of worker processes for some applications running the Bamboo stack.