tag:emulatorwtf.statuspage.io,2005:/historyemulatorwtf Status - Incident History2024-03-26T19:03:55Zemulatorwtftag:emulatorwtf.statuspage.io,2005:Incident/203626842024-03-26T19:03:55Z2024-03-26T19:03:55ZOutputs missing<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>19:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:14</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>15:57</var> UTC</small><br><strong>Investigating</strong> - We're investigating reports of a subset of test run outputs not getting downloaded.</p>tag:emulatorwtf.statuspage.io,2005:Incident/203207612024-03-21T17:22:01Z2024-03-21T17:22:01ZHighly elevated error rates<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>17:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>16:03</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>15:57</var> UTC</small><br><strong>Investigating</strong> - We're experiencing highly elevated error rates in our emulator workers cluster, investigating.</p>tag:emulatorwtf.statuspage.io,2005:Incident/202250572024-03-12T16:43:50Z2024-03-12T16:43:50ZPossible networking issues in emulators<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>13:37</var> UTC</small><br><strong>Investigating</strong> - Our datacenter provider has notified us of a networking issue on their side. This doesn't affect our systems directly but IPv4 connectivity inside the emulators may be degraded.</p>tag:emulatorwtf.statuspage.io,2005:Incident/202082572024-03-10T17:06:56Z2024-03-10T17:06:56ZAPI misbehaving<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>17:06</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>17:04</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>17:04</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Investigating</strong> - We're investigating elevated error rates when trying to run new tests with `ew-cli`.</p>tag:emulatorwtf.statuspage.io,2005:Incident/200177192024-02-19T05:31:52Z2024-02-19T05:51:27ZMissing JUnit result xml files<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>05:31</var> UTC</small><br><strong>Resolved</strong> - A bad release was deployed which broke grabbing test result JUnit XML files. Due to the nature of the bug this flew under our radar in monitoring. We rolled back the release and improved both monitoring and our test coverage to catch this quicker in the future.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>05:29</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>05:21</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:emulatorwtf.statuspage.io,2005:Incident/190449142023-11-07T12:31:53Z2023-11-07T12:31:53ZEmulators outage<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>12:31</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>12:09</var> UTC</small><br><strong>Monitoring</strong> - Emulator workers are coming back online. We're monitoring the situation.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:45</var> UTC</small><br><strong>Identified</strong> - A bad update poisoned our emulator worker pool. Working on bringing the emulator workers back online. ETA for cluster recovery is around 12:15 UTC.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:37</var> UTC</small><br><strong>Investigating</strong> - We are running at reduced capacity, queue times may be long. We are investigating the issue.</p>tag:emulatorwtf.statuspage.io,2005:Incident/188385952023-10-18T10:47:40Z2023-10-18T10:47:41ZVideo recordings missing<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>04:15</var> UTC</small><br><strong>Investigating</strong> - Video recordings are not getting picked up correctly.</p>tag:emulatorwtf.statuspage.io,2005:Incident/175928612023-06-18T13:03:32Z2023-06-18T13:03:32ZEmulator infrastucture upgrade<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>13:03</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>11:44</var> UTC</small><br><strong>Scheduled</strong> - We are rolling out an upgrade across our emulator-serving infrastructure. While we don't expect any issues at all there may be slight hiccups during the process.</p>tag:emulatorwtf.statuspage.io,2005:Incident/173993522023-05-28T17:34:16Z2023-05-28T17:34:16ZNetworking infrastructure maintenance<p><small>May <var data-var='date'>28</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>13:51</var> UTC</small><br><strong>Scheduled</strong> - We will be performing network maintenance on 2023-05-28. While we don't expect any outages there may be a few hiccups during the maintenance period.</p>tag:emulatorwtf.statuspage.io,2005:Incident/172939772023-05-18T18:11:11Z2023-05-18T18:28:50ZTests not triggering properly<p><small>May <var data-var='date'>18</var>, <var data-var='time'>18:11</var> UTC</small><br><strong>Resolved</strong> - Tests did not trigger properly for some customers.</p>tag:emulatorwtf.statuspage.io,2005:Incident/169125812023-04-19T19:01:09Z2023-04-19T19:01:09ZNetworking infrastructure maintenance<p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>19:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Scheduled</strong> - We're making a few changes to our networking infrastructure. While we don't expect any outages there may be a few short disruptions during the process.</p>tag:emulatorwtf.statuspage.io,2005:Incident/165962282023-03-21T20:20:48Z2023-03-21T20:20:48ZAPI unresponsive<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>20:17</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>20:07</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>20:01</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:emulatorwtf.statuspage.io,2005:Incident/164705662023-03-13T07:41:45Z2023-03-13T07:41:45ZVideo recording fails on some emulators<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>07:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>07:40</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>07:27</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>07:14</var> UTC</small><br><strong>Investigating</strong> - We're investigating video recording failing with a pid file error on a number of machines</p>tag:emulatorwtf.statuspage.io,2005:Incident/161807962023-02-17T20:53:59Z2023-02-17T20:54:00ZDatabase maintenance<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>19:31</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>19:24</var> UTC</small><br><strong>Scheduled</strong> - We will be performing some database maintenance where the service may encounter temporary disruptions.</p>tag:emulatorwtf.statuspage.io,2005:Incident/159796742023-01-31T00:42:16Z2023-01-31T00:42:16ZTests not running<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:40</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:36</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:31</var> UTC</small><br><strong>Investigating</strong> - We've got an outage with assigning emulators to incoming tests.</p>tag:emulatorwtf.statuspage.io,2005:Incident/130829632022-11-14T21:00:00Z2022-11-14T21:26:20ZCascading API errors<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - An error in a database migration caused cascading errors in all components, including running tests, checking results & using the main emulator.wtf website.</p>tag:emulatorwtf.statuspage.io,2005:Incident/107073992022-07-28T17:30:58Z2022-07-28T17:30:58ZNetwork maintenance<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>16:02</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>08:47</var> UTC</small><br><strong>Scheduled</strong> - We're changing the networking stack of our emulator infra. While we don't expect any issues there may be degraded performance and increased delays during the maintenance window.</p>tag:emulatorwtf.statuspage.io,2005:Incident/95692612022-03-24T05:15:38Z2022-03-24T05:15:38ZAPI errors<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>05:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>21:42</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Investigating</strong> - We're observing a small number of test shards failing due to API errors</p>tag:emulatorwtf.statuspage.io,2005:Incident/92900802022-02-11T13:34:38Z2022-02-11T13:34:38ZSD card storage errors<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>13:34</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>12:48</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>10:40</var> UTC</small><br><strong>Identified</strong> - We've noticed some storage errors originating from emulators. There may be a slight increase in latency while we resolve the issue.</p>tag:emulatorwtf.statuspage.io,2005:Incident/90854232022-01-14T23:39:16Z2022-01-14T23:39:16ZAPI misbehaving<p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>23:39</var> UTC</small><br><strong>Resolved</strong> - The configuration has been fixed.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Identified</strong> - We've pinpointed the issue to a faulty load-balancer configuration.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>22:54</var> UTC</small><br><strong>Investigating</strong> - API is returning 40x errors.</p>