File Object Lock with WDK

Post by Asaf Shelly
Hi All,
I know that it is possible to have an automatic synchronization that is
http://msdn.microsoft.com/en-us/library/ff544763(VS.85).aspx
Can't find anywhere that says how.
TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.

Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.

-- pa

Asaf Shelly

2010-08-17 22:43:03 UTC

Hi Pavel and Abhishek,

Thanks for the answers. So basically if I need File-Object based
synchronization then I should not use Queue based synchronization and instead
do this manually, correct?

Is there any lock object attached to the File Object?

TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Maxim S. Shatskih

2010-08-17 23:03:59 UTC

Post by Asaf Shelly
Is there any lock object attached to the File Object?

Yes, there is, used to synchronize threads on non-overlapped FO and thus to protect ->CurrentByteOffset from races.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-18 11:29:03 UTC

Hi Maxim,

Thanks. Is this lock something that I can use as a user of KMDF?

Asaf

Post by Asaf Shelly
Is there any lock object attached to the File Object?

Yes, there is, used to synchronize threads on non-overlapped FO and thus to protect ->CurrentByteOffset from races.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Abhishek Ram [MSFT]

2010-08-18 01:51:05 UTC

Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Post by Asaf Shelly
Hi Pavel and Abhishek,
Thanks for the answers. So basically if I need File-Object based
synchronization then I should not use Queue based synchronization and instead
do this manually, correct?
Is there any lock object attached to the File Object?
TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Asaf Shelly

2010-08-18 11:39:03 UTC

Hi Abhishek,

I have a driver which is a higher level to the COM port, so it is using a
Serial Port as the lower driver.
My application opens my driver for communication and the driver can either
read, write, send periodic buffers, and manipulate data. Currently I need an
instance of the driver for every COM port so basically if I have 5 COM ports
on the machine then I also need 5 instances of my virtual device.
What I would like to do is open my device with the COM number so instead of
using \\?\MyDevice1 for COM1 and \\?\MyDevice2 for COM2, I would like to use
\\?\MyDevice\COM1 and \\?\MyDevice\COM2.
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.
I can solve this in many ways but am looking for a clean design with KMDF
support.

From what I have seen there is some support for FOs in KMDF but fewer
samples than any other solution. Is FO support limited in this version of
KMDF?

Thanks for the fast response.

Regards,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Doron Holan [MSFT]

2010-08-18 16:45:14 UTC

you can create your own set of queues per WDFFILEOBJECT. specify the
WDFFILEOBJECT as the parent when you create the queues in
EvtDeviceFileCreate. from your top level dispatching queue, you find the
WDFFILEOBJECT specific queue and forward the request to that queue. you can
then use queue level synchronization and each file object is separate from
each other.

d

"Asaf Shelly" wrote in message news:EEF1F838-3A0A-4788-816C-***@microsoft.com...

Hi Abhishek,

I have a driver which is a higher level to the COM port, so it is using a
Serial Port as the lower driver.
My application opens my driver for communication and the driver can either
read, write, send periodic buffers, and manipulate data. Currently I need an
instance of the driver for every COM port so basically if I have 5 COM ports
on the machine then I also need 5 instances of my virtual device.
What I would like to do is open my device with the COM number so instead of
using \\?\MyDevice1 for COM1 and \\?\MyDevice2 for COM2, I would like to use
\\?\MyDevice\COM1 and \\?\MyDevice\COM2.
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.
I can solve this in many ways but am looking for a clean design with KMDF
support.

From what I have seen there is some support for FOs in KMDF but fewer
samples than any other solution. Is FO support limited in this version of
KMDF?

Thanks for the fast response.

Regards,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Asaf Shelly

2010-08-19 01:38:03 UTC

Hi Doron,

Thanks! This is exactly what I needed. Can you please go over the
documentation and verify that this information is there? I spent a while
searching for samples or this explanation but could not find any. Keep in
mind that it is hard to search for this issue in a search engine. There are
so many search results for "File", "Driver", "Object", "Lock", "Queue" which
are not related to a Driver's File Object based Lock.

Thanks again,
Asaf

Post by Doron Holan [MSFT]
you can create your own set of queues per WDFFILEOBJECT. specify the
WDFFILEOBJECT as the parent when you create the queues in
EvtDeviceFileCreate. from your top level dispatching queue, you find the
WDFFILEOBJECT specific queue and forward the request to that queue. you can
then use queue level synchronization and each file object is separate from
each other.
d
Hi Abhishek,
I have a driver which is a higher level to the COM port, so it is using a
Serial Port as the lower driver.
My application opens my driver for communication and the driver can either
read, write, send periodic buffers, and manipulate data. Currently I need an
instance of the driver for every COM port so basically if I have 5 COM ports
on the machine then I also need 5 instances of my virtual device.
What I would like to do is open my device with the COM number so instead of
using \\?\MyDevice1 for COM1 and \\?\MyDevice2 for COM2, I would like to use
\\?\MyDevice\COM1 and \\?\MyDevice\COM2.
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.
I can solve this in many ways but am looking for a clean design with KMDF
support.
From what I have seen there is some support for FOs in KMDF but fewer
samples than any other solution. Is FO support limited in this version of
KMDF?
Thanks for the fast response.
Regards,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Pavel A.

2010-08-19 07:25:01 UTC

If you open \\?\MyDevice\COM1 and \\?\MyDevice\COM2,
you get handles to different file objects, so synchronous requests
to these run independently.

However, the blocking on sync handles occurs in user mode.
Drivers always complete requests and get out quickly, maybe
with STATUS_PENDING.
Windows is not Unix.

--pa

Post by Asaf Shelly
Hi Doron,
Thanks! This is exactly what I needed. Can you please go over the
documentation and verify that this information is there? I spent a while
searching for samples or this explanation but could not find any. Keep in
mind that it is hard to search for this issue in a search engine. There are
so many search results for "File", "Driver", "Object", "Lock", "Queue" which
are not related to a Driver's File Object based Lock.
Thanks again,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Post by Asaf Shelly
Hi All,
I know that it is possible to have an automatic synchronization
that
is
http://msdn.microsoft.com/en-us/library/ff544763(VS.85).aspx
Can't find anywhere that says how.
TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa

Maxim S. Shatskih

2010-08-19 10:38:35 UTC

Post by Pavel A.
However, the blocking on sync handles occurs in user mode.

In kernel mode in IopSynchronousServiceTail

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Pavel A.

2010-08-19 23:28:25 UTC

Post by Pavel A.
However, the blocking on sync handles occurs in user mode.

In kernel mode in IopSynchronousServiceTail

Ah. Really. It's damn hot here :(
But regarding the app, this is practically same.
-- pa

Post by Maxim S. Shatskih
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com

Asaf Shelly

2010-08-19 20:13:03 UTC

Hi Pavel,

I am forwarding the request to the COM port and unless overlapped IO is used
the thread is suspended until a character is input and the driver remains
locked until the request was completed. I can tell the user to use overlapped
IO but I don't want application to stop responsing if someone will not in the
fufute.

Thanks,
Asaf

Post by Pavel A.
If you open \\?\MyDevice\COM1 and \\?\MyDevice\COM2,
you get handles to different file objects, so synchronous requests
to these run independently.
However, the blocking on sync handles occurs in user mode.
Drivers always complete requests and get out quickly, maybe
with STATUS_PENDING.
Windows is not Unix.
--pa

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Post by Asaf Shelly
Hi All,
I know that it is possible to have an automatic synchronization
that
is
http://msdn.microsoft.com/en-us/library/ff544763(VS.85).aspx
Can't find anywhere that says how.
TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa

Pavel A.

2010-08-19 23:36:16 UTC

Post by Asaf Shelly
Hi Pavel,
I am forwarding the request to the COM port and unless overlapped IO is used
the thread is suspended until a character is input and the driver remains
locked until the request was completed. I can tell the user to use overlapped
IO but I don't want application to stop responsing if someone will not in the
fufute.
Thanks,
Asaf

Try to open a new handle for each app thread -
each thread will then wait on its own file object, independently from each
other.

Regards,
-- pa

Post by Doron Holan [MSFT]
you can create your own set of queues per WDFFILEOBJECT. specify the
WDFFILEOBJECT as the parent when you create the queues in
EvtDeviceFileCreate. from your top level dispatching queue, you find the
WDFFILEOBJECT specific queue and forward the request to that queue.
you
can
then use queue level synchronization and each file object is separate from
each other.
d
Hi Abhishek,
I have a driver which is a higher level to the COM port, so it is using a
Serial Port as the lower driver.
My application opens my driver for communication and the driver can either
read, write, send periodic buffers, and manipulate data. Currently I
need
an
instance of the driver for every COM port so basically if I have 5 COM ports
on the machine then I also need 5 instances of my virtual device.
What I would like to do is open my device with the COM number so
instead
of
using \\?\MyDevice1 for COM1 and \\?\MyDevice2 for COM2, I would like
to
use
\\?\MyDevice\COM1 and \\?\MyDevice\COM2.
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.
I can solve this in many ways but am looking for a clean design with KMDF
support.
From what I have seen there is some support for FOs in KMDF but fewer
samples than any other solution. Is FO support limited in this version of
KMDF?
Thanks for the fast response.
Regards,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Post by Asaf Shelly
Hi Pavel and Abhishek,
Thanks for the answers. So basically if I need File-Object based
synchronization then I should not use Queue based synchronization
and
instead
do this manually, correct?
Is there any lock object attached to the File Object?
TIA,
Asaf

Post by Asaf Shelly
Hi All,
I know that it is possible to have an automatic synchronization
that
is
http://msdn.microsoft.com/en-us/library/ff544763(VS.85).aspx
Can't find anywhere that says how.
TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain
device
object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa

Asaf Shelly

2010-08-23 07:37:03 UTC

Hi Pavel,

This is exactly the plan, I only want the driver to be protected from
mistakes. The only way to communicate with a COM port is to open it once and
use a background reader thread on the same file handle. It is a mistake
waiting to happen.

Regards,
Asaf

Try to open a new handle for each app thread -
each thread will then wait on its own file object, independently from each
other.
Regards,
-- pa

Post by Doron Holan [MSFT]
you can create your own set of queues per WDFFILEOBJECT. specify the
WDFFILEOBJECT as the parent when you create the queues in
EvtDeviceFileCreate. from your top level dispatching queue, you find the
WDFFILEOBJECT specific queue and forward the request to that queue.
you
can
then use queue level synchronization and each file object is separate from
each other.
d
Hi Abhishek,
I have a driver which is a higher level to the COM port, so it is using a
Serial Port as the lower driver.
My application opens my driver for communication and the driver can either
read, write, send periodic buffers, and manipulate data. Currently I
need
an
instance of the driver for every COM port so basically if I have 5 COM ports
on the machine then I also need 5 instances of my virtual device.
What I would like to do is open my device with the COM number so
instead
of
using \\?\MyDevice1 for COM1 and \\?\MyDevice2 for COM2, I would like
to
use
\\?\MyDevice\COM1 and \\?\MyDevice\COM2.
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.
I can solve this in many ways but am looking for a clean design with KMDF
support.
From what I have seen there is some support for FOs in KMDF but fewer
samples than any other solution. Is FO support limited in this version of
KMDF?
Thanks for the fast response.
Regards,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file
object?

Post by Asaf Shelly
Hi Pavel and Abhishek,
Thanks for the answers. So basically if I need File-Object based
synchronization then I should not use Queue based synchronization
and
instead
do this manually, correct?
Is there any lock object attached to the File Object?
TIA,
Asaf

Post by Asaf Shelly
Hi All,
I know that it is possible to have an automatic synchronization
that
is
http://msdn.microsoft.com/en-us/library/ff544763(VS.85).aspx
Can't find anywhere that says how.
TIA,
Asaf

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain
device
object,
can be synchronized with other callbacks of that file object, and
with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa

Abhishek Ram [MSFT]

2010-08-18 18:51:19 UTC

Post by Asaf Shelly
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.

Are you saying your driver does not receive any other request for that
device object? Or is it for that file object?
What synchronization scope did you set in the case that you observed this?
Also does the application use FILE_FLAG_OVERLAPPED when it opens the handle?

Post by Asaf Shelly
Hi Abhishek,
I have a driver which is a higher level to the COM port, so it is using a
Serial Port as the lower driver.
My application opens my driver for communication and the driver can either
read, write, send periodic buffers, and manipulate data. Currently I need an
instance of the driver for every COM port so basically if I have 5 COM ports
on the machine then I also need 5 instances of my virtual device.
What I would like to do is open my device with the COM number so instead of
using \\?\MyDevice1 for COM1 and \\?\MyDevice2 for COM2, I would like to use
\\?\MyDevice\COM1 and \\?\MyDevice\COM2.
This is working, except that if my device is forwading a request in
transparent mode then it is possible that the lower level driver will block
to completion and my driver cannot handle any other request till completion.
I can solve this in many ways but am looking for a clean design with KMDF
support.
From what I have seen there is some support for FOs in KMDF but fewer
samples than any other solution. Is FO support limited in this version of
KMDF?
Thanks for the fast response.
Regards,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Asaf Shelly

2010-08-19 01:46:03 UTC

Hi Abhishek,

My test application does not use FILE_FLAG_OVERLAPPED because my user is not
used to it.
The device might receive other requests, mainly Create File / Close Handle /
Cleanup, and power, so basically I do need device level synchronization as
well.
Currently the synchronization scope is Queue level, but this without doron's
solution is only device wide lock.

Honestly I would expect WDF to forward the request to the File Object
specific queue automatically.

Thanks for all the help,
Asaf

Post by Abhishek Ram [MSFT]
Can you provide a little more information on what you are trying to
accomplish? Which callbacks do you need to synchronize based on file object?

Automatic synchronization is not _based_ on file objects.
Rather, callbacks of a file object, that belongs to certain device object,
can be synchronized with other callbacks of that file object, and with
other callbacks of that device object.
This is implemented by taking a lock in the device object.
Call WdfDeviceInitSetFileObjectConfig
with WDF_OBJECT_ATTRIBUTES where you specify SynchronizationScope
and ExecutionLevel as needed.
-- pa
.

Maxim S. Shatskih

2010-08-19 10:51:05 UTC

Post by Asaf Shelly
My test application does not use FILE_FLAG_OVERLAPPED because my user is not
used to it.
The device might receive other requests, mainly Create File / Close Handle /
Cleanup, and power, so basically I do need device level synchronization as
well.
Currently the synchronization scope is Queue level, but this without doron's
solution is only device wide lock.

So what? You need delice-level synchronization, and the KMDF queue gives you this. All is OK.

And, if you need file-level synchronization, then just never use FILE_FLAG_OVERLAPPED, and all IRPs sent to this file object will be serialized by IO.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-19 20:57:03 UTC

I need driver-wide synchronization to protect FO creation and cleanup.
On the other hand I don't want an application using ReadFile on a COM port
to block the entire driver from working. At least block only the application
that did the read, not all the applications using the device.

My real scenario is that I am keeping an internal read queue and the
application reads periodically all the collected data (I know that there is
always data to read).

Generally speaking I can see a scenario where the application is creating a
file and using a reader thread and a writer thread on the same file handle. I
would want to protect driver wide resources but allow parallel reads and
writes. What is the implementation for this? consider old school fopen("COM1")

Asaf

So what? You need delice-level synchronization, and the KMDF queue gives you this. All is OK.
And, if you need file-level synchronization, then just never use FILE_FLAG_OVERLAPPED, and all IRPs sent to this file object will be serialized by IO.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Maxim S. Shatskih

2010-08-19 21:03:46 UTC

Post by Asaf Shelly
On the other hand I don't want an application using ReadFile on a COM port
to block the entire driver from working

Sorry, but COM ports usually support only 1 file object created, the second create will fail.

This is done with Exclusive flag in IoCreateDevice.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-23 07:37:03 UTC

My driver will open the COM port just once but the User application is
talking to the COM port via my driver..

Post by Asaf Shelly
On the other hand I don't want an application using ReadFile on a COM port
to block the entire driver from working

Sorry, but COM ports usually support only 1 file object created, the second create will fail.
This is done with Exclusive flag in IoCreateDevice.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Abhishek Ram [MSFT]

2010-08-19 22:01:42 UTC

Post by Asaf Shelly
Generally speaking I can see a scenario where the application is creating a
file and using a reader thread and a writer thread on the same file handle. I
would want to protect driver wide resources but allow parallel reads and
writes. What is the implementation for this? consider old school fopen("COM1")

If your application wants to have parallel reads and writes on the same file
handle, it needs to specify FILE_FLAG_OVERLAPPED. Please see the
documentation for CreateFile, specifically the documentation of this flag:

"If this flag is specified, the file can be used for simultaneous read and
write operations.
If this flag is not specified, then I/O operations are serialized, even if
the calls to the read and write functions specify an OVERLAPPED structure."

Post by Asaf Shelly
I need driver-wide synchronization to protect FO creation and cleanup.
On the other hand I don't want an application using ReadFile on a COM port
to block the entire driver from working. At least block only the application
that did the read, not all the applications using the device.
My real scenario is that I am keeping an internal read queue and the
application reads periodically all the collected data (I know that there is
always data to read).
Generally speaking I can see a scenario where the application is creating a
file and using a reader thread and a writer thread on the same file handle. I
would want to protect driver wide resources but allow parallel reads and
writes. What is the implementation for this? consider old school fopen("COM1")
Asaf

So what? You need delice-level synchronization, and the KMDF queue gives
you this. All is OK.
And, if you need file-level synchronization, then just never use
FILE_FLAG_OVERLAPPED, and all IRPs sent to this file object will be
serialized by IO.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Pavel A.

2010-08-19 23:26:39 UTC

If your application wants to have parallel reads and writes on the same
file handle, it needs to specify FILE_FLAG_OVERLAPPED.

What if he opens a *new* handle (creating a new file object)
for each application thread? Then threads won't block each other?
Of course, the driver will need to allow multiple creates coming from one
app.

-- pa

Post by Abhishek Ram [MSFT]
Please see the documentation for CreateFile, specifically the documentation
"If this flag is specified, the file can be used for simultaneous read and
write operations.
If this flag is not specified, then I/O operations are serialized, even if
the calls to the read and write functions specify an OVERLAPPED structure."

Post by Asaf Shelly
My test application does not use FILE_FLAG_OVERLAPPED because my user is not
used to it.
The device might receive other requests, mainly Create File / Close Handle /
Cleanup, and power, so basically I do need device level
synchronization as
well.
Currently the synchronization scope is Queue level, but this without doron's
solution is only device wide lock.

So what? You need delice-level synchronization, and the KMDF queue gives
you this. All is OK.
And, if you need file-level synchronization, then just never use
FILE_FLAG_OVERLAPPED, and all IRPs sent to this file object will be
serialized by IO.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Abhishek Ram [MSFT]

2010-08-20 00:04:53 UTC

Yes, if the threads were to use different file handles then they can do I/O
simultaneously even without FILE_FLAG_OVERLAPPED. However, as you point out,
the device should allow multiple handles to be opened to it. I believe
serial ports are exclusive access devices, so they allow no more than one
handle to be opened to them (as Maxim also mentioned in another post).

If your application wants to have parallel reads and writes on the same
file handle, it needs to specify FILE_FLAG_OVERLAPPED.

Post by Abhishek Ram [MSFT]
Please see the documentation for CreateFile, specifically the
"If this flag is specified, the file can be used for simultaneous read
and write operations.
If this flag is not specified, then I/O operations are serialized, even
if the calls to the read and write functions specify an OVERLAPPED
structure."

Post by Asaf Shelly
My test application does not use FILE_FLAG_OVERLAPPED because my user is not
used to it.
The device might receive other requests, mainly Create File / Close Handle /
Cleanup, and power, so basically I do need device level
synchronization as
well.
Currently the synchronization scope is Queue level, but this without doron's
solution is only device wide lock.

So what? You need delice-level synchronization, and the KMDF queue
gives you this. All is OK.
And, if you need file-level synchronization, then just never use
FILE_FLAG_OVERLAPPED, and all IRPs sent to this file object will be
serialized by IO.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Asaf Shelly

2010-08-23 07:48:03 UTC

This is my problem exactly. COM Port are opened once and used with multiple
threads using blocking operations (ReadFile, WriteFile without overlapped).
It is most probable that one of the users of this driver will use the same
method when integrating in an old application that is currently using the COM
port directly.

The correct method is to use overlapped IO but I still think that the KMDF
library should provide support for File Object isolation. Currently the
scenario is that if I am using any out-of-the-box synchronization I am
risking a situation of a single user application deadlocking the driver. For
example if an application is using the driver to open a COM port which is not
connected to a cable and then uses ReadFile on that handle then the lower
level Serial driver will block to completion and would practically deny any
communication from any user application.

What I am looking for is not a solution to the problem, I am looking for a
way to isolate the rest of the driver from the 'stupid user'.

Thanks,
Asaf

Post by Abhishek Ram [MSFT]
Yes, if the threads were to use different file handles then they can do I/O
simultaneously even without FILE_FLAG_OVERLAPPED. However, as you point out,
the device should allow multiple handles to be opened to it. I believe
serial ports are exclusive access devices, so they allow no more than one
handle to be opened to them (as Maxim also mentioned in another post).

If your application wants to have parallel reads and writes on the same
file handle, it needs to specify FILE_FLAG_OVERLAPPED.

Post by Asaf Shelly
My test application does not use FILE_FLAG_OVERLAPPED because my user is not
used to it.
The device might receive other requests, mainly Create File / Close Handle /
Cleanup, and power, so basically I do need device level
synchronization as
well.
Currently the synchronization scope is Queue level, but this without doron's
solution is only device wide lock.

So what? You need delice-level synchronization, and the KMDF queue
gives you this. All is OK.
And, if you need file-level synchronization, then just never use
FILE_FLAG_OVERLAPPED, and all IRPs sent to this file object will be
serialized by IO.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Pavel A.

2010-08-23 11:30:08 UTC

"Asaf Shelly" <***@Shelly.co.il> wrote in message news:2CEB94A2-FFAA-45B9-93E8-***@microsoft.com...
..........

Post by Asaf Shelly
What I am looking for is not a solution to the problem, I am looking for a
way to isolate the rest of the driver from the 'stupid user'.

Not sure if I completely understand this statement, but maybe you need
a better interface concept.

See http://www.osronline.com/showThread.cfm?link=188211
message #26

"you must never, ever, not even once, think that ReadFile, WriteFile, and/or
DeviceIoControl
are the *user visible* interfaces to your driver. Do this, and it warps
your thinking
into trying to make this low-level interface "easy to use". No, decide what
you want the upper-level user interface to look like, to the application
programmer. Then writer a library that translates that into low-level calls
like DeviceIoControl. ...

There is a line between the effort we expend in kernel space and the effort
we expend in user space to make a piece of hardware work. The user space
gets split between what the application writer *has* to see, and what you
can write as a library. The smaller the kernel piece, the faster you get it
done, the more robust it is likely to be. .............
So you design the API *first*, then figure out how you are going to map it
to
low-level calls, then figure out how you are going to get the performance
you need "
( Joseph Newcomer, co-author of the "Developing Windows NT Device Drivers"
book)

IMHO, "stupid users" is a misnomer. Our users are not stupid at all ;)
Instead, how about "users that don't want to care about boring
details" ?
They are not going to access low level interfaces at all, exactly because
these are encumbered with lot of boring details.
These folks should use higher level, more robust interfaces, like message
queues,
and automatic memory management.

Regards,
--pa

Asaf Shelly

2010-08-23 22:42:03 UTC

Hi Pavel,

I understand and agree with you. Here is my scenario:

There used to be an old physical box connected to the COM port and managed
by it. This box was used to manage a frame based communication bus with
accurate timing that applications failed to provide. Some 10 years ago I
created a driver which was a replacement for the box. The solution was a
filter driver attaching on to the COM port and emulating the external box
with all serial commands. This was a good solution becuase there was no need
to modify the existing applications using the old box.
Today we had the opportumity to redesign the system. In order to allow
backward compatibility I allow transparent Read and Write to COM port until a
special io control is received.

The problem I am seeing today is more of a design concept problem. I don't
see a reason why a single application could make my driver not respond to any
other application. This can be solved with some work but I would expect KMDF
to provide some protection from such mistakes. Eventually there will be some
programmer who will not consider this situation and release a driver that can
be non responsive to all client applications because the lower level driver
is blocking.

Asaf

Post by Pavel A.
..........

Post by Asaf Shelly
What I am looking for is not a solution to the problem, I am looking for a
way to isolate the rest of the driver from the 'stupid user'.

Not sure if I completely understand this statement, but maybe you need
a better interface concept.
See http://www.osronline.com/showThread.cfm?link=188211
message #26
"you must never, ever, not even once, think that ReadFile, WriteFile, and/or
DeviceIoControl
are the *user visible* interfaces to your driver. Do this, and it warps
your thinking
into trying to make this low-level interface "easy to use". No, decide what
you want the upper-level user interface to look like, to the application
programmer. Then writer a library that translates that into low-level calls
like DeviceIoControl. ...
There is a line between the effort we expend in kernel space and the effort
we expend in user space to make a piece of hardware work. The user space
gets split between what the application writer *has* to see, and what you
can write as a library. The smaller the kernel piece, the faster you get it
done, the more robust it is likely to be. .............
So you design the API *first*, then figure out how you are going to map it
to
low-level calls, then figure out how you are going to get the performance
you need "
( Joseph Newcomer, co-author of the "Developing Windows NT Device Drivers"
book)
IMHO, "stupid users" is a misnomer. Our users are not stupid at all ;)
Instead, how about "users that don't want to care about boring
details" ?
They are not going to access low level interfaces at all, exactly because
these are encumbered with lot of boring details.
These folks should use higher level, more robust interfaces, like message
queues,
and automatic memory management.
Regards,
--pa

Pavel A.

2010-08-24 02:55:02 UTC

Post by Asaf Shelly
Hi Pavel,
There used to be an old physical box connected to the COM port and managed
by it. This box was used to manage a frame based communication bus with
accurate timing that applications failed to provide. Some 10 years ago I
created a driver which was a replacement for the box. The solution was a
filter driver attaching on to the COM port and emulating the external box
with all serial commands. This was a good solution becuase there was no need
to modify the existing applications using the old box.
Today we had the opportumity to redesign the system. In order to allow
backward compatibility I allow transparent Read and Write to COM port until a
special io control is received.
The problem I am seeing today is more of a design concept problem. I don't
see a reason why a single application could make my driver not respond to any
other application. This can be solved with some work but I would expect KMDF
to provide some protection from such mistakes. Eventually there will be some
programmer who will not consider this situation and release a driver that can
be non responsive to all client applications because the lower level driver
is blocking.
Asaf

OK, so the problem is that some app can wait on the real COM port,
and then you cannot send the special ioctl because you cannot open
another handle to the device, because it is exclusive.

Various solutions are possible here, and can be done in WDM or KMDF,
but I cannot agree that KMDF can change anything in this situation; it is
just a wrapper around bare OS (a.k.a. WDM).

Regards,
-- pa

Asaf Shelly

2010-08-27 05:18:06 UTC

Post by Pavel A.
OK, so the problem is that some app can wait on the real COM port,
and then you cannot send the special ioctl because you cannot open
another handle to the device, because it is exclusive.
Various solutions are possible here, and can be done in WDM or KMDF,
but I cannot agree that KMDF can change anything in this situation; it is
just a wrapper around bare OS (a.k.a. WDM).

The problem I am having is that the synchronization is done by the KMDF
engine.
Serial is always blocking reads but what if someone is using a driver, for
example a disk driver, which is not blocking unless the queue is full? The
problem I am seeing will appear only at deployment. Bad timing for a driver
redesing.

Asaf

Pavel A.

2010-08-27 15:30:19 UTC

KMDF (and the win. kernel) also are around for quite long. Not a good time
to redesign these as well...

Why you cannot kill the app holding on the com port as Maxim suggested?

-- pa

Asaf Shelly

2010-08-28 22:00:03 UTC

Post by Pavel A.
KMDF (and the win. kernel) also are around for quite long. Not a good time
to redesign these as well...
Why you cannot kill the app holding on the com port as Maxim suggested?

Because it is wrong to ask the user to close an application with penalty of
entire system not responding.
What if my driver is managing a global resource? Can you imagine having to
ask the users to locate an close the Bluetooth application which is making my
driver irresponsive? Until you find that service you cannot access any
Bluetooth functionality because the driver is locked...

Asaf

Pavel A.

2010-08-28 22:49:42 UTC

Post by Pavel A.
KMDF (and the win. kernel) also are around for quite long. Not a good time
to redesign these as well...
Why you cannot kill the app holding on the com port as Maxim suggested?

I *can* imagine asking the user to reboot.
Users know that Windows likes reboots ;)

--pa

Maxim S. Shatskih

2010-08-29 13:07:23 UTC

Post by Pavel A.
I *can* imagine asking the user to reboot.
Users know that Windows likes reboots ;)

There is nothing about the entire system, only about 1 app (or service).

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-29 21:12:07 UTC

Post by Pavel A.
I *can* imagine asking the user to reboot.
Users know that Windows likes reboots ;)
--pa

I have to admit that I just restarted my machine because of the compiler
issue.

If I get a single call asking me to come see why "my driver isn't working",
I'll have to go over the configuration and then the user's code etc. A good
day's work and I can respond. Imagine buying Microsoft webcam and it will
stall because your mobile phone service is also using WinUSB.. Everytime you
have a call your camera will miss frames.

A device designed for handling File Objects is expecting to handle multiple
users otherwise what's the point in FOs?

Asaf

Maxim S. Shatskih

2010-08-29 21:21:25 UTC

Post by Asaf Shelly
A device designed for handling File Objects is expecting to handle multiple
users

...and serial port is expecting to handle 1 user only.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-30 06:08:09 UTC

Post by Asaf Shelly
A device designed for handling File Objects is expecting to handle multiple
users

....and serial port is expecting to handle 1 user only.

This is why I found the design bug. Any other driver would delay your upper
layer driver only in extreme cases, making it hard to detect or understand

Maxim S. Shatskih

2010-08-29 13:06:47 UTC

Post by Asaf Shelly
Because it is wrong to ask the user to close an application with penalty of
entire system not responding.

Serial is an exclusive device by its very design, so, such things are OK.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-29 21:15:03 UTC

Post by Asaf Shelly
Because it is wrong to ask the user to close an application with penalty of
entire system not responding.

Serial is an exclusive device by its very design, so, such things are OK.

I am allowing general file-system targets including COM, LPT, Pipes, etc.
My user does not care how I implement my 'engine'. Just as you don't care
that IRDA is over a serial port and USB is over PCI...

Maxim S. Shatskih

2010-08-29 21:22:02 UTC

Post by Asaf Shelly
I am allowing general file-system targets including COM, LPT, Pipes, etc.

COM and LPT are not file systems.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-30 06:12:06 UTC

Post by Asaf Shelly
I am allowing general file-system targets including COM, LPT, Pipes, etc.

COM and LPT are not file systems.

FS as in "FS" Object Store, not NTFS. Don't blame me that the world is using
a File System to access drivers.

Maxim S. Shatskih

2010-08-29 13:06:03 UTC

Post by Asaf Shelly
Serial is always blocking reads but what if someone is using a driver, for
example a disk driver, which is not blocking unless the queue is full?

Disk stack is never ever blocking.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-30 06:10:06 UTC

Post by Asaf Shelly
Serial is always blocking reads but what if someone is using a driver, for
example a disk driver, which is not blocking unless the queue is full?

Disk stack is never ever blocking.

Unless the CD has errors on it..

Maxim S. Shatskih

2010-08-30 09:06:16 UTC

Post by Maxim S. Shatskih
Disk stack is never ever blocking.

Unless the CD has errors on it..

This is only on old ATA controllers where CD retries stall the HD hanging off the same PATA cable.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Maxim S. Shatskih

2010-08-24 19:41:56 UTC

Post by Asaf Shelly
The problem I am seeing today is more of a design concept problem. I don't
see a reason why a single application could make my driver not respond to any
other application.

So what? just kill the app and go on.

OS-provided serial.sys has no such protection.

--
Maxim S. Shatskih
Windows DDK MVP
***@storagecraft.com
http://www.storagecraft.com

Asaf Shelly

2010-08-27 05:14:06 UTC

Serial solved this by allow only one device access at a time, even though (if
I remember correctly) it will allow a write operation while read is still
blocking.

Asaf

Post by Asaf Shelly
The problem I am seeing today is more of a design concept problem. I don't
see a reason why a single application could make my driver not respond to any
other application.

So what? just kill the app and go on.
OS-provided serial.sys has no such protection.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Pavel A.

2010-08-30 23:07:02 UTC

This problem most likely has a reasonable solution, but unfortunately you're
looking in a wrong place.
The serial port driver won't change, WDF design won't change.
Playing word games - FO or not FO - won't help.
The reasonabe solution is balanced somewhere in the middle, between user's
ideal and what you are ready to provide.
--pa

Post by Asaf Shelly
Serial solved this by allow only one device access at a time, even though (if
I remember correctly) it will allow a write operation while read is still
blocking.
Asaf

Post by Asaf Shelly
The problem I am seeing today is more of a design concept problem. I don't
see a reason why a single application could make my driver not respond to any
other application.

So what? just kill the app and go on.
OS-provided serial.sys has no such protection.
--
Maxim S. Shatskih
Windows DDK MVP
http://www.storagecraft.com
.

Abhishek Ram [MSFT]

2010-08-17 03:06:06 UTC