Discussion:
[win-pv-devel] Hi, a question about the checksum offload and WinDivert
Haohao Lee
2018-11-06 08:33:38 UTC
Permalink
Hi Xen folks,

Background:
I am a Windows application developer. We developed an application which
modifies packets and rejects them back into the network stack to do some
network proxy transparently.

We achieved this by using WinDivert (https://reqrypt.org/windivert.html)
that is a tool/driver allows user-mode applications to capture/modify/drop
network packets sent to/from the Windows network stack.

Our app worked well on physical Windows machines but on Xen virtual
machines we encountered a problem.

Problem:
- Everything works well before we start our app.
- Network traffic is blocked after the app is started, even a single SYN
packet couldn't be sent out.

Workaround:
If we disable the checksum offload in Xen Net Driver, everything starts to
work again.

Testing Environment:
Xen Virtual Machine: Windows 7 Sp1 x64 with latest updates
Xen Net Driver: Driver version 3.0.144.590 xennet.sys version 2.3.0.144

I have a couple of questions:
1. Is this a problem of WinDivert driver or Xen Net driver from your
perspective?
2. If this belongs to Xen Net driver, does the latest driver fix this?
3. I found many articles on the Internet which teach people to disable
checksum offload (and other kinds of offload) for Xen virtual machines,
e.g. some tutorials from AWS. Why is this option ON by default if it
shouldn't be, or is there any introduction about the context why it is ON
by default? I know what TCP/IP checksum is, but in virtual machine context,
I have no idea if it is necessary or not.

Any comment or suggestion is appreciated.

Thanks

Hao
Paul Durrant
2018-11-06 11:06:30 UTC
Permalink
De-htmling...

My responses indented:

---
From: win-pv-devel [mailto:win-pv-devel-***@lists.xenproject.org] On Behalf Of Haohao Lee
Sent: 06 November 2018 08:34
To: win-pv-***@lists.xenproject.org
Subject: [win-pv-devel] Hi, a question about the checksum offload and WinDivert

Hi Xen folks,

Background:
I am a Windows application developer. We developed an application which modifies packets and rejects them back into the network stack to do some network proxy transparently.

We achieved this by using WinDivert (https://reqrypt.org/windivert.html) that is a tool/driver allows user-mode applications to capture/modify/drop network packets sent to/from the Windows network stack.

Our app worked well on physical Windows machines but on Xen virtual machines we encountered a problem.

Problem:
- Everything works well before we start our app.
- Network traffic is blocked after the app is started, even a single SYN packet couldn't be sent out.

Workaround:
If we disable the checksum offload in Xen Net Driver, everything starts to work again.
Hi Hao,
Which checksum offload? Just TCP or IPv4 too?
Testing Environment:
Xen Virtual Machine: Windows 7 Sp1 x64 with latest updates
Xen Net Driver: Driver version 3.0.144.590 xennet.sys version 2.3.0.144

I have a couple of questions:
1. Is this a problem of WinDivert driver or Xen Net driver from your perspective?
Since the stable (i.e. 8.x) drivers pass all logo tests (which have detailed checks of the semantics of checksum offload, LSO, etc.) the I'd say the problem lies in the application. One thing to try is disabling LRO though.. this is disabled for logo testing since the version of NDIS we use doesn't actually support it. (Moving to a newer NDIS is on the TODO list).
2. If this belongs to Xen Net driver, does the latest driver fix this?
3. I found many articles on the Internet which teach people to disable checksum offload (and other kinds of offload) for Xen virtual machines, e.g. some tutorials from AWS. Why is this option ON by default if it shouldn't be, or is there any introduction about the context why it is ON by default? I know what TCP/IP checksum is, but in virtual machine context, I have no idea if it is necessary or not.

Any comment or suggestion is appreciated.
The answer is "it's complicated" :-) The default set of offloads is the set we use in Citrix branded versions of the drivers for XenServer and, whilst there used to be many issues with such offloads in the past (pre Xenserver 7.x), we have not had *any* reports from the field to suggest there are any current issues with checksum or large packet offloads in the 8.x drivers.
Cheers,
Paul
Thanks

Hao
Haohao Lee
2018-11-06 16:01:17 UTC
Permalink
Post by Paul Durrant
De-htmling...
---
Sent: 06 November 2018 08:34
Subject: [win-pv-devel] Hi, a question about the checksum offload and WinDivert
Hi Xen folks,
I am a Windows application developer. We developed an application which modifies packets and rejects them back into the network stack to do some network proxy transparently.
We achieved this by using WinDivert (https://reqrypt.org/windivert.html) that is a tool/driver allows user-mode applications to capture/modify/drop network packets sent to/from the Windows network stack.
Our app worked well on physical Windows machines but on Xen virtual machines we encountered a problem.
- Everything works well before we start our app.
- Network traffic is blocked after the app is started, even a single SYN packet couldn't be sent out.
If we disable the checksum offload in Xen Net Driver, everything starts to work again.
Hi Hao,
Which checksum offload? Just TCP or IPv4 too?
The driver property does not mention IP or TCP, just checksum offload (But our problem only occurs for TCP)
Here I have a screenshot Loading Image...
Xen Virtual Machine: Windows 7 Sp1 x64 with latest updates
Xen Net Driver: Driver version 3.0.144.590 xennet.sys version 2.3.0.144
1. Is this a problem of WinDivert driver or Xen Net driver from your perspective?
Since the stable (i.e. 8.x) drivers pass all logo tests (which have detailed checks of the semantics of checksum offload, LSO, etc.) the I'd say the problem lies in the application. One thing to try is disabling LRO though.. this is disabled for logo testing since the version of NDIS we use doesn't actually support it. (Moving to a newer NDIS is on the TODO list).
New findings, If I disable checksum offload, the average throughput goes up from 600Mbit/s to 1Gbit/s as well. Therefore now I suspect there may be something wrong with this option ON.
2. If this belongs to Xen Net driver, does the latest driver fix this?
3. I found many articles on the Internet which teach people to disable checksum offload (and other kinds of offload) for Xen virtual machines, e.g. some tutorials from AWS. Why is this option ON by default if it shouldn't be, or is there any introduction about the context why it is ON by default? I know what TCP/IP checksum is, but in virtual machine context, I have no idea if it is necessary or not.
Any comment or suggestion is appreciated.
The answer is "it's complicated" :-) The default set of offloads is the set we use in Citrix branded versions of the drivers for XenServer and, whilst there used to be many issues with such offloads in the past (pre Xenserver 7.x), we have not had *any* reports from the field to suggest there are any current issues with checksum or large packet offloads in the 8.x drivers.
By the way, it seems I cannot install 8.x Xen Net Driver directly. The OS keeps telling me my old driver is the best one.
Cheers,
Paul
Thanks
Hao
Paul Durrant
2018-11-06 16:10:25 UTC
Permalink
-----Original Message-----
Behalf Of Haohao Lee
Sent: 06 November 2018 16:01
Subject: Re: [win-pv-devel] Hi, a question about the checksum offload and
WinDivert
Post by Paul Durrant
De-htmling...
---
Behalf Of Haohao Lee
Post by Paul Durrant
Sent: 06 November 2018 08:34
Subject: [win-pv-devel] Hi, a question about the checksum offload and
WinDivert
Post by Paul Durrant
Hi Xen folks,
I am a Windows application developer. We developed an application which
modifies packets and rejects them back into the network stack to do some
network proxy transparently.
Post by Paul Durrant
We achieved this by using WinDivert (https://reqrypt.org/windivert.html)
that is a tool/driver allows user-mode applications to capture/modify/drop
network packets sent to/from the Windows network stack.
Post by Paul Durrant
Our app worked well on physical Windows machines but on Xen virtual
machines we encountered a problem.
Post by Paul Durrant
- Everything works well before we start our app.
- Network traffic is blocked after the app is started, even a single SYN
packet couldn't be sent out.
Post by Paul Durrant
If we disable the checksum offload in Xen Net Driver, everything starts
to work again.
Post by Paul Durrant
Hi Hao,
Which checksum offload? Just TCP or IPv4 too?
The driver property does not mention IP or TCP, just checksum
offload (But our problem only occurs for TCP)
Post by Paul Durrant
Here I have a screenshot https://user-
images.githubusercontent.com/238419/47770931-46b6ad80-dd1c-11e8-8010-
42d05baf018b.png
Ok, that tells me you are not using Xen Project PV drivers. Where did you get your PV drivers from?

Paul
Post by Paul Durrant
Xen Virtual Machine: Windows 7 Sp1 x64 with latest updates
Xen Net Driver: Driver version 3.0.144.590 xennet.sys version 2.3.0.144
1. Is this a problem of WinDivert driver or Xen Net driver from your
perspective?
Post by Paul Durrant
Since the stable (i.e. 8.x) drivers pass all logo tests (which have
detailed checks of the semantics of checksum offload, LSO, etc.) the I'd
say the problem lies in the application. One thing to try is disabling LRO
though.. this is disabled for logo testing since the version of NDIS we
use doesn't actually support it. (Moving to a newer NDIS is on the TODO
list).
Post by Paul Durrant
New findings, If I disable checksum offload, the average throughput
goes up from 600Mbit/s to 1Gbit/s as well. Therefore now I suspect there
may be something wrong with this option ON.
Post by Paul Durrant
2. If this belongs to Xen Net driver, does the latest driver fix this?
3. I found many articles on the Internet which teach people to disable
checksum offload (and other kinds of offload) for Xen virtual machines,
e.g. some tutorials from AWS. Why is this option ON by default if it
shouldn't be, or is there any introduction about the context why it is ON
by default? I know what TCP/IP checksum is, but in virtual machine
context, I have no idea if it is necessary or not.
Post by Paul Durrant
Any comment or suggestion is appreciated.
The answer is "it's complicated" :-) The default set of offloads is
the set we use in Citrix branded versions of the drivers for XenServer
and, whilst there used to be many issues with such offloads in the past
(pre Xenserver 7.x), we have not had *any* reports from the field to
suggest there are any current issues with checksum or large packet
offloads in the 8.x drivers.
Post by Paul Durrant
By the way, it seems I cannot install 8.x Xen Net Driver directly.
The OS keeps telling me my old driver is the best one.
Post by Paul Durrant
Cheers,
Paul
Thanks
Hao
_______________________________________________
win-pv-devel mailing list
https://lists.xenproject.org/mailman/listinfo/win-pv-devel
Haohao Lee
2018-11-06 18:20:53 UTC
Permalink
Post by Paul Durrant
-----Original Message-----
Behalf Of Haohao Lee
Sent: 06 November 2018 16:01
Subject: Re: [win-pv-devel] Hi, a question about the checksum offload and
WinDivert
Post by Paul Durrant
De-htmling...
---
Behalf Of Haohao Lee
Post by Paul Durrant
Sent: 06 November 2018 08:34
Subject: [win-pv-devel] Hi, a question about the checksum offload and
WinDivert
Post by Paul Durrant
Hi Xen folks,
I am a Windows application developer. We developed an application which
modifies packets and rejects them back into the network stack to do some
network proxy transparently.
Post by Paul Durrant
We achieved this by using WinDivert (
https://reqrypt.org/windivert.html)
Post by Paul Durrant
that is a tool/driver allows user-mode applications to
capture/modify/drop
Post by Paul Durrant
network packets sent to/from the Windows network stack.
Post by Paul Durrant
Our app worked well on physical Windows machines but on Xen virtual
machines we encountered a problem.
Post by Paul Durrant
- Everything works well before we start our app.
- Network traffic is blocked after the app is started, even a single SYN
packet couldn't be sent out.
Post by Paul Durrant
If we disable the checksum offload in Xen Net Driver, everything starts
to work again.
Post by Paul Durrant
Hi Hao,
Which checksum offload? Just TCP or IPv4 too?
The driver property does not mention IP or TCP, just checksum
offload (But our problem only occurs for TCP)
Post by Paul Durrant
Here I have a screenshot https://user-
images.githubusercontent.com/238419/47770931-46b6ad80-dd1c-11e8-8010-
42d05baf018b.png
Ok, that tells me you are not using Xen Project PV drivers. Where did you
get your PV drivers from?
Post by Paul Durrant
It should be from Citrix Virtual Desktop solution. Here is a screenshot
of the driver version
Loading Image...
Post by Paul Durrant
Can I use Xen Project PV drivers for Citrix Products for they are still
Xen Server based anyway?
Post by Paul Durrant
Thanks for you kind response so far.
Paul
Post by Paul Durrant
Xen Virtual Machine: Windows 7 Sp1 x64 with latest updates
Xen Net Driver: Driver version 3.0.144.590 xennet.sys version 2.3.0.144
1. Is this a problem of WinDivert driver or Xen Net driver from your
perspective?
Post by Paul Durrant
Since the stable (i.e. 8.x) drivers pass all logo tests (which have
detailed checks of the semantics of checksum offload, LSO, etc.) the I'd
say the problem lies in the application. One thing to try is disabling LRO
though.. this is disabled for logo testing since the version of NDIS we
use doesn't actually support it. (Moving to a newer NDIS is on the TODO
list).
Post by Paul Durrant
New findings, If I disable checksum offload, the average throughput
goes up from 600Mbit/s to 1Gbit/s as well. Therefore now I suspect there
may be something wrong with this option ON.
Post by Paul Durrant
2. If this belongs to Xen Net driver, does the latest driver fix this?
3. I found many articles on the Internet which teach people to disable
checksum offload (and other kinds of offload) for Xen virtual machines,
e.g. some tutorials from AWS. Why is this option ON by default if it
shouldn't be, or is there any introduction about the context why it is ON
by default? I know what TCP/IP checksum is, but in virtual machine
context, I have no idea if it is necessary or not.
Post by Paul Durrant
Any comment or suggestion is appreciated.
The answer is "it's complicated" :-) The default set of offloads is
the set we use in Citrix branded versions of the drivers for XenServer
and, whilst there used to be many issues with such offloads in the past
(pre Xenserver 7.x), we have not had *any* reports from the field to
suggest there are any current issues with checksum or large packet
offloads in the 8.x drivers.
Post by Paul Durrant
By the way, it seems I cannot install 8.x Xen Net Driver directly.
The OS keeps telling me my old driver is the best one.
Post by Paul Durrant
Cheers,
Paul
Thanks
Hao
_______________________________________________
win-pv-devel mailing list
https://lists.xenproject.org/mailman/listinfo/win-pv-devel
Paul Durrant
2018-11-07 09:40:28 UTC
Permalink
De-html-ing (again)...

From: Haohao Lee [mailto:***@gmail.com]
Sent: 06 November 2018 18:21
To: Paul Durrant <***@citrix.com>
Cc: win-pv-***@lists.xenproject.org
Subject: Re: [win-pv-devel] Hi, a question about the checksum offload and WinDivert

[snip]
Post by Paul Durrant
The driver property does not mention IP or TCP, just checksum
offload (But our problem only occurs for TCP)
Here I have a screenshot https://user-
http://images.githubusercontent.com/238419/47770931-46b6ad80-dd1c-11e8-8010-
42d05baf018b.png
Ok, that tells me you are not using Xen Project PV drivers. Where did you get your PV drivers from?
It should be from Citrix Virtual Desktop solution. Here is a screenshot of the driver version
 https://user-images.githubusercontent.com/238419/47770995-8087b400-dd1c-11e8-894b-9fd0c2188732.png
No, those drivers are nothing to do with Citrix. They are a set of drivers written for Windows guests running on upstream Xen with source released under the GPL, making them impossible to get logo signed by Microsoft... so unusable for Citrix customers in general. See https://wiki.xen.org/wiki/Xen_FAQ_Drivers,_Windows.
Post by Paul Durrant
Can I use Xen Project PV drivers for Citrix Products for they are still Xen Server based anyway?
The Xen Project PV drivers are indeed derived from drivers written by myself and my colleagues at Citrix. They were originally just for XenServer but we made them open source (with BSD license to avoid logo signing issues) and they are now designed to work well on a vanilla upstream Xen installation as well as under XenServer, so you should have no problem using them in either environment. If you are running XenServer though then you really have no reason not to use the Citrix branded version of the drivers that come on the installation media.

For more information see the project front page at https://xenproject.org/developers/teams/windows-pv-drivers.html and you can find the latest set of (Linux Foundation) signed drivers at http://xenbits.xen.org/pvdrivers/win/8.2.1.
Post by Paul Durrant
Thanks for you kind response so far. 
No problem. Please note that neither the Xen Project or XenServer drivers are compatible with the GPLPV drivers so you will need to completely purge those drivers from your VM (and return to using emulated hardware) *before* attempting to install new drivers.

Cheers,

Paul

Loading...