History of Events
9th October 2024 - Microsoft "Patch Tuesday" has caused a number of issues
Microsoft "Patch Tuesday" has caused a number of issues whereby Windows Updates were either getting stuck, or causing issues.
Replication Delays and reboots occurred during service hours, however customer services remained mostly operational despite short interuptions for each device/service
Server loads were higher than usual for a few hours as the effects were mitigated and managed
VPN Connections were seen to fail. Reconnection/restarts of client machines was required
14th - 16th August 2024 - 3rd Party Service Failure affecting Development Office
On Wednesday 14th August 2024 a fault was found with a 3rd party publishing software service. After contacting their support multiple times it was determined the supplier could no longer be trusted to honour the licences provided and purchased by us.
A decision was taken to immediately invoke our contingency plan and cease the use of their software service, remove all their software from our machines and develop a replacement service to ensure our developers could continue to work.
This problem caused a delay to our development and delivery teams until Friday 16th August at 6pm, whereby we had replaced the old software with our own solution.
For more information please see our Latest News Post from August 2024
Developers: Please uninstall DSX Studio 5.3 and all 3rd party publishing software and replace with DSX Studio 5.4 - See Incident Reference: 2024081610000017 for more information
Following successful replacement, the supplier and their software has also been banned from any future use by any developer or contractor delivering any service on behalf of CodeStep UK in order to protect the supply chain of our customers
17th June 2024 - Network Scanning Service Upgrade
Our Scanning Service has been changed to dramatically reduce the traffic, improve deployment to new monitoring stations and servers with a new managed setup process, and increase the number of scan types available. The new service will be tested over the coming days and now monitors upcoming OS updates, unapproved changes by Windows Update as well as provide operators with an Automated Update tool to simplify future upgrades. The deployment of this service caused some unintended errors codes to be displayed which have subsequanetly cleared as the job list was processed. Over 200 individual scans are now performed on our networks to ensure reliability and early warnings of issues
20th May 2024 - Network Scanning Service Generating Incorrect Failure Codes
Due to an OS upgrade on one of our scanning servers, network devices were falling into a low-power state that caused scanning services to incorrectly report errors across all monitored services. Whilst all hosting and customer services were up and running, this status page showed errors from Environment Faults to Hardware Faults intermittently.
Configuration changes have been applied to mitigate the low-power state issue and tests will run for a couple of days to ensure the problem is rectified
15th May 2024 - OEM Service Update caused minor outage
An overnight OEM system update caused an outage in DC2 resulting in networking and other services failing to communicate.
Customers using SaaS Services bound to DC2 may have been unable to login for a short duration from 8.30am until approx 10:12am whilst services recovered.
1st May 2024 - DSX 5.3 Rollout Begins
As part of our continuous upgrade cycle, all Core Services, WebDesk Pro and WebDesk SiteMaker systems were upgraded to DSX 5.3. This resulted in a momentary loss of service across each application as the update was applied.
Customer SaaS Services will be upgraded over the coming days/weeks in accordance with customer maintenance bookings
15th February 2024 - New Automated Monitoring and Auditing Services
As part of our ISO Accreditation application and business practices, we have now updated our monitoring solutions so all services are scanned on a more frequent basis.
Customer services are scanned every minute for availability, and we have increased our vulnerability scanning using our new monitoring service.
In line with our "Simple Service" ethos - this service page now only shows a simple report - and automatically expands with more detailed information in the event of a fault, maintenance window or other event
20th January 2024 - New Ticket System for Customer Support Services
We have deployed a new ticket management suite to enable customers to email us using a dedicated (per-product) email address that will feed directly into our support team's ticket management system. Customers do not need to login to a seperate portal - simply continue to use your favourite email client and send emails as usual.
All Partners and Resellers have been sent a setup pack that allows you to integrate your own email services directly - enabling direct support with your own chosen email addresses/domains
All direct customers will receive instructions on using a new email address - please ensure you direct all support queries to the new address to ensure our entire team can respond as quickly as possible.
10th January 2024 - Critical Microsoft CVE Patches Applied
At approx 17.15hrs Microsoft KB5034273 was automatically applied to mitigate threats from CVE-2023-36042, CVE-2023-0056, CVE-2023-0057 and CVE-2023-21312
This led to a momentary loss of service to some affected servers whilst services restarted
2nd-16th December 2023 - Datacenter Upgrades and Migration
We have completed the migration of our old datacenter services to new hardware and software platforms.
As well as software upgrades, DC2 now has expanded and upgraded hardware to provide additional services and redundency options. All services now support full replication service across all locations
Stage 1 - Completed 8th December - Routing Services, Charity Services and all Test Services.
Stage 2 - Completed 9th/10th December - Migration of DNS Servers and other non-essential services, UK Dedicated Hosting Services were moved
Stage 3 - Completed 9th December - CodeStep.UK, WebDesk Pro SaaS services and SaaS Licence Services
Stage 4 - Completed 16th December - England-Specific SaaS Services
Stage 5 - Completed 16th December - UK-Specific SaaS Services
Stage 6 - AVAILABLE FROM 8th January 2024 - Operating System Upgrades will be available to any customer with a service contract Free of Charge. Downtime Required.
8th November 2023 - Replication Service Failure
A replication failure occured at 10.05am between DC1 and DC2 following a process fault in the replication service. Services were restarted.
A replication delay of approximately 20 minutes occured whilst services were restarted and re-synchronisation completed. No customer services were affected, and replication has recovered for all Live Services. A Full Services Suite Reboot occured out of hours at 8.50pm to ensure stability.
19th September 2023 - Windows Server Servicing Failure
A servicing failure occured during the installation of Microsoft Update KB5030213 resulting in a rollback of the Active Directory (Europe 2) PDC updates. The Failover Services were immediately activated whilst the primary DC was restored and corrected. No customers were affected during the failover or recovery stages, and all dependent services were re-routed. All services were restored at 11.29am on 20th September.
16th August 2023 - DC3 Hardware Failure Event
A Hardware Failure was detected in DC3 on 16th August 2023. Some services were suspended in DC3 and transferred to the alternate datacenter.
The effects included a failover to the assigned backup DC4, with Active Directory (Europe 2) and associated replication services being re-routed. Customer Services were not affected.
The Donated Charity Services were reduced whilst DC4 handled the additional loads of DC3, with a 77% performance limit applied to ensure the smooth operation of DC4
Following the Hardware Replacement, services were restored and re-routed to the original locations - during which there was a 5 minute delay in replication of customer services during the reversal operation. All events were completed within the SLA period, and no negative customer effects occured.