Category Archives: Outages

This will be for all Outage Notifications

8/17/16 Maintenance Complete

East Kingdom Informational Notification

The East Kingdom Server successfully went through a planned upgrade during the evening
of Tuesday, 8/16/2016 between 11:00 pm and 11:50 pm.

We have expanded the available Server Memory (RAM) from  16 GB to 24 GB which will allow up to further support the needs of the Kingdom.

Thanks & Have a nice day!

Signore Lorenz Greylever

East Kingdom Webministery

4/6/16 Maintenance Complete

East Kingdom Informational Notification

The East Kingdom Server successfully went through a planned upgrade during the evening
of Wednesday, 4/6/2016 between 9:00 pm and 9:50 pm.

“The long and short of it is that we’re upgrading our server – gaining double the memory, 33% more processing power and doubling our hard drive space. Much of that should be trivial, even transparent, but resizing the disk can take a while and can’t be done while the server is in use.

While the everyday impact of this shouldn’t be obvious, it will allow us to add more features and provide more services in general. Behind the scenes, we’ll also experience fewer recurring issues (we’ve been bumping up against memory limitations more and more as time goes on, which sometimes results in database issues that can affect different sites and
applications).
There are some interesting and useful things planned that will take advantage of this upgrade, so keep an eye out :)”
In Service,
Mael Eoin
EK Webministry Special Projects Deputy

 

Thanks & Have a nice day!

Signore Lorenz Greylever

East Kingdom Webministery

4/4/16 – Planned Maintenance

East Kingdom Informational Notification

The East Kingdom Server will undergo a planned upgrade during the late evening of Wednesday, 4/6/2016.


This is the third in a series of planned updates this year. (The second update did not require an interruption of services, and was performed last Thursday.)


We will be migrating from our current level of service “Linode 8GB” to the next tier of service “Linode 16GB” with the same Hosting service, which will appear to be seamless to the populace. This will be expanding our available resources and offer a better user experience.

Time Frame 9:00pm-12:30am Eastern Time

This is expected to run for 144 minutes. (I am giving it extra time in case anything happens and a recover is required.)

During this time frame all East Kingdom services will be unavailable to the populace.

Tools that backbone or pull from the server will also be affected (IE: Email, Help Desk, List Server, etc).

Frequently asked questions about this update:

What
is this?

We are increasing our current level of service to the next available Tier from our current Host provider.

How
much faster?
  • Service Level..Linode 8GB *..Linode 16GB **
  • RAM………………..8 GB……………….16 GB
  • CPU…………………6 Cores…………..8 Cores
  • Storage……………192 GB SSD….384 GB SSD
  • Transfer…………..8 TB………………..16 TB
  • Network In……..40 Gbps………….40 Gbps
  • Network Out…. 1000 Mbps…….2000 Mbps

* Current Service Level

** Upgraded Service level

We are doubling our available RAM, Storage, Transfer amount and Outward Bandwidth. 2 Additional CPU will also be allocated for our use.

Will
this break stuff?

Upgrading an existing deployment to a new and larger plan should not affect the current state of the information on your Linode. This will happen automatically, until you resize the new disks.

How it works 1. Linode is shut down and migrated
You will experience downtime while your Linode is migrated. We estimate 144 minutes to migrate your Linode, but that may vary based on host and network load.2. Enjoyment
After the migration completes, you can take advantage of the new resources by resizing your disks.

My apologies for the short notice, however due to the current memory usage of the server, this needs to be done quickly. Please spread the word.

Thanks & Have a nice day!

Signore Lorenz Greylever

East Kingdom Webministery

3/30/16 East Kingdom Wiki

EAST KINGDOM   Populace Alert
Alert! From: The Webministry
   

INITIAL ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (HIGH) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase in contact volume.
     
Tool/Service: East Kingdom Wiki
Impacted Areas: wiki.eastkingdom.org
Description: We have uncovered a coding issue with an upgrade of the Wiki server that does not work with non-English characters, causing pages to “disappear” and were not accessible to the Populace.

All attempts to fix the Latin-1 vs UTF-8 coding differences were unsuccessful and we have had to revert to a previous database backup prior to the upgrade, which is 5 days old.

Ticket Number: None.
Severity: HIGH
Event Start/Finish: 3-25-16 8pm  / 3-30-16 11:30pm (approximate)
     
Please Do Not Open Service Tickets For This Issue

 

This rollback has unfortunately removed any updates made over the past 5 days. Please check your pages for missing information.

We apologize for this inconvenience.

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

   
     

4/20/15 Maintenance Outage

EAST KINGDOM   Populace Alert
Alert! From: The Webministry
   

INITIAL ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (HIGH) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase in contact volume.
     
Tool/Service: Websites Affected: All 
Impacted Areas: eastkingdom.org
Description: We are planning to update the East Kingdom server to better serve the populace at large and increase the capabilities of our offered services.The outage time frame is planned for 4/20/2015 from approximately 1am to 7am.A six-hour window is allocated, however the actual downtime can be much less.

All websites and services will be down during this period of updates.

There is no expected fallout for individual groups or websites.

Ticket Number: None.
Severity: HIGH
Event Start/Finish: 4-20-15 1am  / 3-20-15 7am (approximate)
     
Please Do Not Open Service Tickets For This Issue

 

This update will expand the capabilities of the server beyond its current configuration and bring us in line with our host’s current package offerings without any increase in our current payment schedule.

Please spread the word that all services will not be accessible for the duration of this planned maintenance.

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

   
     

3/7/2015 Maintenance Outage

EAST KINGDOM   Populace Alert
Alert! From: The Webministry
   

INITIAL ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (HIGH) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase in contact volume.
     
Tool/Service: Websites Affected: All 
Impacted Areas: eastkingdom.org
Description: We have been informed that our host will be restarting our server on the following date and time:

* 2015-03-07 9:00:00 AM EST *

A two-hour window is allocated, however the actual downtime can be much less.

Ticket Number: None.
Severity: HIGH
Event Start/Finish: 3-07-15 9am  / 3-07-15 12:00pm (approximate)
     
Please Do Not Open Service Tickets For This Issue

 

These updates are required to protect the security and safe operations of not only our infrastructure, but yours as well. We understand that a disruption with such limited notice is inconvenient, and we hope you understand that these measures are warranted due to the severity of the XSAs (Xen Security Advisories).

Unfortunately, due the logistical demands of this effort, your assigned windows are not changeable and the host reboots are mandatory.

Please spread the word that all services will not be accessible for the duration of their planned maintenance.

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

   
     

1/28/2015 Maintenance Outage

EAST
KINGDOM
Populace Alert
Alert! From: The Webministry

INITIAL ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (Low) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase in contact volume.
Tool/Service: Websites Affected: *.eastkingdom.org that use a Database.

Additional Services: NONE.

Impacted Areas: All Database Driven Websites on the Domain
Description: Around 11:00am everyone was unable to access DB driven services for the East Kingdom server.
Ticket Number: None.
Severity: Low
Event Start/Finish: 1-28-15 11am / 1-28-15 12:30pm
Please Do Not Open Service Tickets For This Issue

 

In the continuing upgrades to the East Kingdom server and services, the web server was restarted to remove a potential security vulnerability.

The restart of the SQL server took longer than expected and the service was impacted until restored.

We do not anticipate any further impact from the restart as all services have been tested as performing normally at this time.

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

 

9/18/14 Server Unresponsive

EAST
KINGDOM
Populace Alert
Alert! From: The Webministry

CONTINUING ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (Critical) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase contact volume.
Tool/Service: Websites Affected:

*.eastkingdom.org

Additional Services:

All EK Services

Impacted Areas: All Websites on the EK Domain
Description: Around 9:30am everyone was unable to access all services for the East Kingdom server.

The East Kingdom sever was unresponsive to all attempts of login and maintenance. A service call to our hosting company discovered that the actual hardware was not responding and they had to manually reboot it to get it to recover.

We are awaiting an explanation as to what caused this most recent and the 9/16/14 outage. The connectivity continues to be spotty at best, which is hampering all ability to perform repairs and continue work on the migration off of this service to the new server completely.

As soon as I hear anything, I will post it on the Webministry website (preferred), or directly to Facebook and G+ if it is not responsive.

Ticket Number: None.
Severity: Critical
Event Start/Finish: 9-18-14 9:30am / On Going
Please Do Not Open Service Tickets For This Issue

 

Connectivity to all websites has been restored however it continue to be spotty.

If you cannot connect on the first try, please attempt to connect again by refreshing the web browser or starting a new instance.

We continue to investigate and monitor the server for stability while we work to migrate off the service as quickly as possible. There are 11 websites left to move, and have been problematic during previous attempts. They may have to be moved with errors.

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

East Kingdom

 

9/16/14 Partial Outage

EAST
KINGDOM
Populace Alert
Alert! From: The Webministry

RESOLVED ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (Critical) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase contact volume.
Tool/Service: Websites Affected:

accessibleek.eastkingdom.org
bhakail.eastkingdom.org
et.eastkingdom.org
carillion.eastkingdom.org
eastkingdom.org/equestrian
hartshorn-dale.eastkingdom.org
mobile.eastkingdom.org
oscar.eastkingdom.org
ostgardr.eastkingdom.org
rapier.eastkingdom.org
seneschal.eastkingdom.org
silverrylle.eastkingdom.org
trh.eastkingdom.org
trm.eastkingdom.org
wiki.eastkingdom.org

Additional Services:

EK Calendar

EK Email Boxes (This did not affect Forwarding.)

Impacted Areas: All Websites on the Old Server
Description: The server stopped functioning for an unknown reason around 12:30 am, making multiple websites unavailable to the populace. The server was successfully restarted this morning at approximately 2 am. The Minister of Æther (System Admin) is researching as to the cause of this and will determine preventative measures for the future.
Ticket Number: None.
Severity: Critical
Event Start/Finish: 9-16-14 12:30am / 9-16-14 2:15am
Please Do Not Open Service Tickets For This Issue

 

Connectivity to all websites has been restored. The websites above have data that is about a month old, at this point it appears that there was a ‘rollback’ to a previous backup done by our provider without warning. Investigation is open as to why.

All sites above have lost all updates since that time frame, including the EK Calendar. Luckily, while doing work on trying to upgrade the Calendar service for the kingdom, I had a few off site backups of the database and was able to re-add the following events:

Season’s Beatings Four-Back for More!
A&E (Aethelmearc/East) War
A Market Day at Birka
Winter’s Night Masquerade
Dragonship Haven Baronial Investiture
Feste des Bois
Crown Tourney
East Kingdom 12th Night / Bellringers / Baronial Investiture
La Feste des Glaces
K&Q Rapier Champions
Owlsherst Country 12th Night

And a new event has been successfully added to the calendar:
Quintavia Hafla

Please check for your event on the calendar and also for correct content! If you find any errors in these entries and can not login to fix them yourself due to the recovery, please open up a Support ticket at http://helpdesk.eastkingdom.org and we will fix it for you.

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

East Kingdom

 

9/3/14 Partial Website Outage

 

EAST
KINGDOM
Populace Alert
Alert! From: The Webministry

RESOLVED ALERT NOTIFICATION

Unto the Populace of the East Kingdom, Greetings.
This is being sent to notify you of the (Critical) Severity event(s) below. Because of the possible large impact of such events, this is being documented to avoid an increase contact volume.
Tool/Service: MySQL
Impacted Areas: All DB Driven Websites
Description: The MySQL service on the server stopped for an unknown reason, making multiple websites unavailable to the populace.

The server was successfully restarted this morning. The Minister of Æther (System Admin) is researching as to the cause of this and will determine preventative measures for the future.

Ticket Number: None.
Severity: Critical
Event Start/Finish: 9-2-14 11:30pm / 9-3-14 9:15am
Please Do Not Open Service Tickets For This Issue

As always, if you have any additional questions; concerns or information regarding the above, or any current events or outages, please do not hesitate to contact the Webministry.

In Service to the Kingdom,

The Webministry

East Kingdom