[win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server 2016 - Only Safe Mode working

[Jens Hauser]

Hello List,



I had a Windows Server 2016 as a DomU on a Debian 8.11 now up and running
for about 6 month without any problems. The system contains an Exchange
Server - everything worked smooth.



On last Friday the system seems to have rebooted due to Windows updates
automatically installed. After that I found the server freezed in a screen
with the no longer animated icons.

Another 2012 R2 server was running fine without any problems.



So I tried to find out whats wrong and narrowed it down tot he network
driver as far as I can tell. I found entries in ntbtlog.txt which said that
"AFD.SYS" was not loaded at boot.

The first tries I was unable to boot into safe mode. After a few hickups the
system booted into Safe Mode saying that an upgrade didn't work and that it
was to be rolled back. After that I was able to get into Safe Mode but first
only the one without network drivers.



If I disable Realtek RTL8139 C network card I can boot into normal
operation. With this network card enabled the boot hangs as before.



After installation of Xen Project Win PV drivers 8.2.1 all the necessary
devices where found but a reboot leads tot he Windows screen with the
animated icons floating endlessly.



It all seems to be related to network stuff. There is this Realtek Card
which needs to be disabled and it's not vanishing if I install the PV
drivers - even if the Xen PV Network Device #0 is showing up the Realtek
Card is still there.



Is there any chance to fix this issue? Or do I need to install the server as
a new system?



Kind regards,

Jens

[Jens Hauser]

BTW: This seems to be the same problem:



https://discussions.citrix.com/topic/395149-windows-10-updates-reverts-nic-t
o-realtek-driver/



But I'm afraid I cannot upgrade my Xen so easily as I am pinned to the
stable branch of Debian. Maybe I'll give it a try with Debian 9 .



Kind regards,

Jens





Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im
Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 15:47
An: win-pv-***@lists.xenproject.org
Betreff: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server
2016 - Only Safe Mode working



Hello List,



I had a Windows Server 2016 as a DomU on a Debian 8.11 now up and running
for about 6 month without any problems. The system contains an Exchange
Server - everything worked smooth.



On last Friday the system seems to have rebooted due to Windows updates
automatically installed. After that I found the server freezed in a screen
with the no longer animated icons.

Another 2012 R2 server was running fine without any problems.



So I tried to find out whats wrong and narrowed it down tot he network
driver as far as I can tell. I found entries in ntbtlog.txt which said that
"AFD.SYS" was not loaded at boot.

The first tries I was unable to boot into safe mode. After a few hickups the
system booted into Safe Mode saying that an upgrade didn't work and that it
was to be rolled back. After that I was able to get into Safe Mode but first
only the one without network drivers.



If I disable Realtek RTL8139 C network card I can boot into normal
operation. With this network card enabled the boot hangs as before.



After installation of Xen Project Win PV drivers 8.2.1 all the necessary
devices where found but a reboot leads tot he Windows screen with the
animated icons floating endlessly.



It all seems to be related to network stuff. There is this Realtek Card
which needs to be disabled and it's not vanishing if I install the PV
drivers - even if the Xen PV Network Device #0 is showing up the Realtek
Card is still there.



Is there any chance to fix this issue? Or do I need to install the server as
a new system?



Kind regards,

Jens

[Jens Hauser]

After three days of struggling with the system I finally got it working
again: As a last approach (I should have done that early I guess) I upgraded
the DomHost operating system from Debian 8 to Debian 9 Stretch - and both
Windows Servers started just fine.



I didn't do that before because I thought that Xen 4.4.4 brought with Jessie
is fine enough for current Windows operating systems - well, seems it's not.



Regards,

Jens





Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im
Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 17:34
An: win-pv-***@lists.xenproject.org
Betreff: Re: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows
Server 2016 - Only Safe Mode working



BTW: This seems to be the same problem:



https://discussions.citrix.com/topic/395149-windows-10-updates-reverts-nic-t
o-realtek-driver/



But I'm afraid I cannot upgrade my Xen so easily as I am pinned to the
stable branch of Debian. Maybe I'll give it a try with Debian 9 .



Kind regards,

Jens





Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im
Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 15:47
An: win-pv-***@lists.xenproject.org
Betreff: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server
2016 - Only Safe Mode working



Hello List,



I had a Windows Server 2016 as a DomU on a Debian 8.11 now up and running
for about 6 month without any problems. The system contains an Exchange
Server - everything worked smooth.



On last Friday the system seems to have rebooted due to Windows updates
automatically installed. After that I found the server freezed in a screen
with the no longer animated icons.

Another 2012 R2 server was running fine without any problems.



So I tried to find out whats wrong and narrowed it down tot he network
driver as far as I can tell. I found entries in ntbtlog.txt which said that
"AFD.SYS" was not loaded at boot.

The first tries I was unable to boot into safe mode. After a few hickups the
system booted into Safe Mode saying that an upgrade didn't work and that it
was to be rolled back. After that I was able to get into Safe Mode but first
only the one without network drivers.



If I disable Realtek RTL8139 C network card I can boot into normal
operation. With this network card enabled the boot hangs as before.



After installation of Xen Project Win PV drivers 8.2.1 all the necessary
devices where found but a reboot leads tot he Windows screen with the
animated icons floating endlessly.



It all seems to be related to network stuff. There is this Realtek Card
which needs to be disabled and it's not vanishing if I install the PV
drivers - even if the Xen PV Network Device #0 is showing up the Realtek
Card is still there.



Is there any chance to fix this issue? Or do I need to install the server as
a new system?



Kind regards,

Jens

[Paul Durrant]

Jens,

Glad you got things working again. I suspect you might have run into a kernel bug that Microsoft introduced a while ago and shipped in a Windows update. It causes blue-screens when the PV network driver allocates memory. If this happens during driver bring-up, as a safety net, the behaviour of the system is to fall back to using the emulated network device. A subsequent attempt to enable the PV driver will just hit the same blue-screen. Microsoft have fixed the problem so, as long as your kernel is fully up-to-date, your PV driver should no longer hit the kernel bug.
I'm not aware of any compatibility issues with Xen 4.4.4, or older Debian dom0 kernels, but 4.4.4 is very old now and has been out of security support for a while so updating to a newer hypervisor and dom0 kernel is a good move anyway.

Cheers,

Paul

From: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] On Behalf Of Jens Hauser
Sent: 18 November 2018 17:39
To: win-pv-***@lists.xenproject.org
Subject: Re: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server 2016 - Only Safe Mode working

After three days of struggling with the system I finally got it working again: As a last approach (I should have done that early I guess) I upgraded the DomHost operating system from Debian 8 to Debian 9 Stretch - and both Windows Servers started just fine.

I didn't do that before because I thought that Xen 4.4.4 brought with Jessie is fine enough for current Windows operating systems - well, seems it's not.

Regards,
Jens


Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 17:34
An: win-pv-***@lists.xenproject.org<mailto:win-pv-***@lists.xenproject.org>
Betreff: Re: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server 2016 - Only Safe Mode working

BTW: This seems to be the same problem:

https://discussions.citrix.com/topic/395149-windows-10-updates-reverts-nic-to-realtek-driver/

But I'm afraid I cannot upgrade my Xen so easily as I am pinned to the stable branch of Debian. Maybe I'll give it a try with Debian 9 ...

Kind regards,
Jens


Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 15:47
An: win-pv-***@lists.xenproject.org<mailto:win-pv-***@lists.xenproject.org>
Betreff: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server 2016 - Only Safe Mode working

Hello List,

I had a Windows Server 2016 as a DomU on a Debian 8.11 now up and running for about 6 month without any problems. The system contains an Exchange Server - everything worked smooth.

On last Friday the system seems to have rebooted due to Windows updates automatically installed. After that I found the server freezed in a screen with the no longer animated icons.
Another 2012 R2 server was running fine without any problems.

So I tried to find out whats wrong and narrowed it down tot he network driver as far as I can tell. I found entries in ntbtlog.txt which said that "AFD.SYS" was not loaded at boot.
The first tries I was unable to boot into safe mode. After a few hickups the system booted into Safe Mode saying that an upgrade didn't work and that it was to be rolled back. After that I was able to get into Safe Mode but first only the one without network drivers.

If I disable Realtek RTL8139 C network card I can boot into normal operation. With this network card enabled the boot hangs as before.

After installation of Xen Project Win PV drivers 8.2.1 all the necessary devices where found but a reboot leads tot he Windows screen with the animated icons floating endlessly.

It all seems to be related to network stuff. There is this Realtek Card which needs to be disabled and it's not vanishing if I install the PV drivers - even if the Xen PV Network Device #0 is showing up the Realtek Card is still there.

Is there any chance to fix this issue? Or do I need to install the server as a new system?

Kind regards,
Jens

[Jens Hauser]

Dear Paul,



thank you for your explanation although I think I never saw a blue-screen
the last three days. It was just that the virtual Windows kind of hangs in
two different kinds. The first one (with only the RTL8139 C device active)
freezes the system with no more rotating dots while booting. CPU usage fell
to "0". The second one (after installation of current Windows PV drivers)
kind of freezes with dots still rotating. CPU consumption always changes
around 32% while 4 cores available to the virtual system (xl top view).



I had those blue-screens when I was using gplpv drivers about a year ago
after updating operating system.



Xen 4.4.4 is the current version of Debian Jessie - the LTS version which I
normally prefer for production systems. Your hint concerning security
support is kind of frightening . I will have a look which of our Dom0 are
still using this Debian version and upgrade soon.



Kind regards,

Jens





Von: Paul Durrant [mailto:***@citrix.com]
Gesendet: Montag, 19. November 2018 10:27
An: 'Jens Hauser'; win-pv-***@lists.xenproject.org
Betreff: RE: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows
Server 2016 - Only Safe Mode working



Jens,



Glad you got things working again. I suspect you might have run into a
kernel bug that Microsoft introduced a while ago and shipped in a Windows
update. It causes blue-screens when the PV network driver allocates memory.
If this happens during driver bring-up, as a safety net, the behaviour of
the system is to fall back to using the emulated network device. A
subsequent attempt to enable the PV driver will just hit the same
blue-screen. Microsoft have fixed the problem so, as long as your kernel is
fully up-to-date, your PV driver should no longer hit the kernel bug.

I'm not aware of any compatibility issues with Xen 4.4.4, or older Debian
dom0 kernels, but 4.4.4 is very old now and has been out of security support
for a while so updating to a newer hypervisor and dom0 kernel is a good move
anyway.



Cheers,



Paul



From: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] On
Behalf Of Jens Hauser
Sent: 18 November 2018 17:39
To: win-pv-***@lists.xenproject.org
Subject: Re: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows
Server 2016 - Only Safe Mode working



After three days of struggling with the system I finally got it working
again: As a last approach (I should have done that early I guess) I upgraded
the DomHost operating system from Debian 8 to Debian 9 Stretch - and both
Windows Servers started just fine.



I didn't do that before because I thought that Xen 4.4.4 brought with Jessie
is fine enough for current Windows operating systems - well, seems it's not.



Regards,

Jens





Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im
Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 17:34
An: win-pv-***@lists.xenproject.org
Betreff: Re: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows
Server 2016 - Only Safe Mode working



BTW: This seems to be the same problem:



https://discussions.citrix.com/topic/395149-windows-10-updates-reverts-nic-t
o-realtek-driver/



But I'm afraid I cannot upgrade my Xen so easily as I am pinned to the
stable branch of Debian. Maybe I'll give it a try with Debian 9 .



Kind regards,

Jens





Von: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] Im
Auftrag von Jens Hauser
Gesendet: Sonntag, 18. November 2018 15:47
An: win-pv-***@lists.xenproject.org
Betreff: [win-pv-devel] Xen 4.4.4 + Win PV Drivers 8.2.1 + Windows Server
2016 - Only Safe Mode working



Hello List,



I had a Windows Server 2016 as a DomU on a Debian 8.11 now up and running
for about 6 month without any problems. The system contains an Exchange
Server - everything worked smooth.



On last Friday the system seems to have rebooted due to Windows updates
automatically installed. After that I found the server freezed in a screen
with the no longer animated icons.

Another 2012 R2 server was running fine without any problems.



So I tried to find out whats wrong and narrowed it down tot he network
driver as far as I can tell. I found entries in ntbtlog.txt which said that
"AFD.SYS" was not loaded at boot.

The first tries I was unable to boot into safe mode. After a few hickups the
system booted into Safe Mode saying that an upgrade didn't work and that it
was to be rolled back. After that I was able to get into Safe Mode but first
only the one without network drivers.



If I disable Realtek RTL8139 C network card I can boot into normal
operation. With this network card enabled the boot hangs as before.



After installation of Xen Project Win PV drivers 8.2.1 all the necessary
devices where found but a reboot leads tot he Windows screen with the
animated icons floating endlessly.



It all seems to be related to network stuff. There is this Realtek Card
which needs to be disabled and it's not vanishing if I install the PV
drivers - even if the Xen PV Network Device #0 is showing up the Realtek
Card is still there.



Is there any chance to fix this issue? Or do I need to install the server as
a new system?



Kind regards,

Jens