libusb_claim_interface fails on Mac OS with HID devices

[northcob]

Software that uses libusb on the current 10.6.6 Mac OS always fails if the device reports containing a HID. It seems a number of devices do this so that the HID driver loads automatically on Mac OS and Windows. That way the device manufacturer's software people don't need to bother to write a driver.

The cause is that libusb_claim_interface fails.

This happens even when run as root AND the recommended dummy kext is installed.

This is from pwwws trying to access a Fine Offset WH1080 weather station through Pyusb and libusb 1.0.8

bash-3.2# python TestWeatherStation?.py
Traceback (most recent call last):

File "TestWeatherStation?.py", line 94, in <module>

sys.exit(main())

File "TestWeatherStation?.py", line 56, in main

ws = WeatherStation?.weather_station()

File "/Users/Shared/Public/Weather?/pywws-10.10_r298/pywws/WeatherStation.py", line 205, in init

raise IOError("Claim interface failed")

IOError: Claim interface failed
bash-3.2#

If someone can give me a clue about how to extract debugging information, I am happy to do it.

Bill Northcott

[hjelmn]

Not a bug. Use hidapi or a similar library to connect to HID devices.

[northcob]

A bug is when software fails to perform as described in the documentation.

Your documentation clearly states that libusb is compatible with Mac OS X. So libusb_claim_interface should work as documented on Mac OS X. It does not. So it is a bug.

If you can't be bothered to fix it, which is up to you. Then fix the documentation and stop claiming that libusb works on Mac OS X.

The current situation just wastes a lot of time.

[stuge]

libusb is very much compatible with Mac OS X, but libusb is not suitable for programming every type of USB device.

Just as you would not likely use libusb to access a USB memory stick (you would probably use the kernel filesystem API instead) you should not use libusb for HID class devices.

Nathan is the Mac OS X expert in libusb, but it seems to me that Apple have stopped the previous workaround that allowed the kernel HID driver to be blocked for specific USB devices (codeless kext) from working since a recent system update, so at the moment there is indeed no way to access HID class devices using libusb, which isn't the fault of libusb.

Please see ​http://libusb.org/wiki/FAQ#CanIcreateadriverlessdeviceusingHIDclass for more information and a link to HIDAPI.

[northcob]

Replying to stuge:

libusb is very much compatible with Mac OS X, but libusb is not suitable for programming every type of USB device.

It may be compatible, as in it compiles and runs, but a documented function fails to work which constitutes a bug.

Just as you would not likely use libusb to access a USB memory stick (you would probably use the kernel filesystem API instead) you should not use libusb for HID class devices.

This is all very well if one is writing some code to access a specific device. The problem is that libusb still fails the 'does what it says on the tin' test. To quote your overview page "Linux and Darwin (Mac OS X) are supported in the latest release. Windows support is present in git as of August 2010." There no caveats about HID devices being a problem.

The issue here is that because libusb advertises itself as supporting all these systems it has been used by the writers of language bindings such as pyusb and ruby-usb. Indeed if you see my original bug description it used pyusb. Googling around, I even found a couple of HID bindings for Python both of which use libusb. Even hidapi can use libusb on Linux, but does not pretend that it would work on Windows or Mac OS.

Nathan is the Mac OS X expert in libusb, but it seems to me that Apple have stopped the previous workaround that allowed the kernel >HID driver to be blocked for specific USB devices (codeless kext) from working since a recent system update, so at the moment there is >indeed no way to access HID class devices using libusb, which isn't the fault of libusb.

When the software does not work as documented it is clearly the fault of libusb. If you fix either the software or the documentation, then it will become a feature not a bug. It is quite simple you cold save a huge amount of other people's time by just adding a caveat about HID devices to your Wiki front page.

Is that really too painful?

I am afraid I don't have enough understanding of USB to fix this myself, but reading Apple's documentation it seems to me that it might be possible to have code in the libusb shield kext which prevented the device reporting its HID interface. OTOH this might not be possible on restart because Mac hardware builds a device tree during the POST using the OpenFirmware? stuff.

Please see ​http://libusb.org/wiki/FAQ#CanIcreateadriverlessdeviceusingHIDclass for more information and a link to HIDAPI.

I had actually fund that after much Googling but it is less than clear unless one already understands the problem.

[hjelmn]

Replying to northcob:

It may be compatible, as in it compiles and runs, but a documented function fails to work which constitutes a bug.

I don't see how a documented function has failed to work? The OS has claimed the device and there is no way to punt the driver. This is a documented limitation in the darwin implementation.

This is all very well if one is writing some code to access a specific device. The problem is that libusb still fails the 'does what it says on the tin' test. To quote your overview page "Linux and Darwin (Mac OS X) are supported in the latest release. Windows support is present in git as of August 2010." There no caveats about HID devices being a problem.

The caveat is that libusb can not possibly access interfaces that are claimed by a kernel driver if the kernel driver can't be punted. Its just the reality of working with hardware from userspace. This is true for any number of devices classes: i.e CDC, HUB, HID...

The issue here is that because libusb advertises itself as supporting all these systems it has been used by the writers of language bindings such as pyusb and ruby-usb. Indeed if you see my original bug description it used pyusb. Googling around, I even found a couple of HID bindings for Python both of which use libusb. Even hidapi can use libusb on Linux, but does not pretend that it would work on Windows or Mac OS.

hidapi can use libusb on Linux but uses the native HID support in OSX and Windows. No need to use libusb.

When the software does not work as documented it is clearly the fault of libusb. If you fix either the software or the documentation, then it will become a feature not a bug. It is quite simple you cold save a huge amount of other people's time by just adding a caveat about HID devices to your Wiki front page.

Is that really too painful?

I agree that we should probably make it a little more clear that hid devices should not be accessed using libusb.

I am afraid I don't have enough understanding of USB to fix this myself, but reading Apple's documentation it seems to me that it might be possible to have code in the libusb shield kext which prevented the device reporting its HID interface. OTOH this might not be possible on restart because Mac hardware builds a device tree during the POST using the OpenFirmware? stuff.

No need to use a shield kext. Use hidapi when working with hid devices. Use libusb for devices that don't abuse the HID class.

Anyway, this ticket is a duplicate of #33 and I still do see no reason to add IOHID support to libusb.

[northcob]

Replying to hjelmn:

As it seems you have no intention of being helpful to your users, I will just move on to somewhere where I can use my time more productively.

Bill

[stockli]

Wow, these are pretty strongly worded discussions. Please don't forget that libusb is a open source project managed by freelancers. If there is a possible solution out there, it will hopefully be found, but maybe not tomorrow or next week.

I have a few comments to add which may help to move forward:

• Please try to rolling back to the iousbfamily kext of 10.6.6 to 10.6.4. I have seen the MultiBeast? tool (see: ​http://tonymacx86.blogspot.com/), and you may also download the debugging versions of earlier usbfamily kext's (​http://developer.apple.com/hardwaredrivers/download/usbdebug.html). Not sure if it helps, but it may be worth a try.

• northcob, please inquire with the Apple Developer Mailing list about how to write a codeless kernel extension for 10.6.6 that correctly seizes your device. After all, Apple provides tutorials for such codeless kext's, and they might be interested to provide updates to them so that these tutorials continue working after 10.6.6.

• the library structure of libusb-1.0 is capable to forcefully detach a kernel driver. This is currently only implemented for linux. Maybe either Nathan or someone else with good Darwin coding capabilities will be capable of implementing this feature also for Darwin.

• next a remark with usb versus hid: since any hid device is simply a usb device, I disagree with the comments above that libusb should not be able to handle hid devices. libusb can handle all hid devices and it correctly does so. Open it, initialize it, write to it, read from it, close it. This is how I access the OpenCockpits? hardware interfaces. It is pretty simple to handle HID devices as USB devices. No need for a HID abstraction layer.

• northcob: I now have the same problem as you. My HID devices on OSX are also not working any more with 10.6.6 and libusb on OSX. They work fine on linux with the libusb_detach_kernel_driver() call even when the device had been captured by the linux HID driver. We will find a solution for the HID devices to be working on OSX. For now, try rolling back your IOUSBfamily kext.

After all, and finally: thanks a lot for this great project, libusb-1.0. It really is well done and please continue the HQ software engineering.

Reto

[xiaofan]

If you read through the libusb mailing list, actually there are some debates on this topic. Unfortunately Peter Stuge (the project admin) and Nathan (the main Mac OS X libusb developer) are opposite to the idea of adding the HID backend to Mac OS X.

Thread:
​http://libusb.6.n5.nabble.com/Re-libusb-33-Mac-OS-X-libusb-1-0-native-HID-backend-td3345660.html

[stockli]

Well, I absolutely agree with both Peter and Nathan that there should not be HID support in libusb (neither for OSX, linux nor for Windows). There are enough HID backends from each system out there.

The whole story for OSX is whether we can continue to write user-space USB drivers for our HID devices. Remember, a HID device is simply a USB device. That requires to either a) not load or b) detach the OSX HID driver from the device.

a) used to work on OSX by use of a codeless kext
b) works fine on linux by use of the libusb_detach_kernel_driver() call in libusb.

We might proceed with trying to get either a) working again on OSX (I have submitted a bug report to Apple), or implementing b) for OSX like it works on Linux.

Reto

[xiaofan]

If you are content with a) then probably there will be a way out. As for b), as far as I know it is not going to work. There is no such facility available. Many users are not content with a) actually.

[hjelmn]

As Xiofan says it is impossible to punt a kernel driver in OS X. To add this functionality it would require changes to IOUSBDevice and IOUSBLib. Its unlikely Apple would accept these changes for 10.5+ and impossible for anything older :(.

The best way to write userspace drivers for hid devices is to use hidapi. It is cross platform and you can avoid having to punt the kernel hid driver. This includes Linux when hidraw is available.

[stockli]

All:

Thanks for mentioning hidraw, I did not even know about it. Checked it out just now. This is definitely the code that will make libusb-1.0 obsolete for accessing HID devices on OSX, and hopefully soon on Linux as well.

I will ASAP try a parallel implementation of hidraw and libusb in my own usb device driver code to access my HID devices and see how they perform. It looks like the hidraw API is very similar to the libusb API.

So in summary: the use of hidraw fixes
a) the codeless kext failure in OSX 10.6.6
b) the need to put the kernel HID driver in OSX.

Problem solved.

Reto

[stuge]

Replying to stockli:

Thanks for mentioning hidraw

Note that hidraw is just the Linux-specific API to access HID class devices.

We recommend using ​HIDAPI from Signal 11 Software, as mentioned on ​http://libusb.org/wiki/FAQ#CanIcreateadriverlessdeviceusingHIDclass, to access HID class devices.

[stockli]

Sorry, I actually meant HIDAPI :-)

Sorry for the confusion

[northcob]

Replying to stockli:

Wow, these are pretty strongly worded discussions. Please don't forget that libusb is a open source project managed by freelancers. If there is a possible solution out there, it will hopefully be found, but maybe not tomorrow or next week.

I was a bit pissed off because the essentially undocumented issues with libusb on Darwin wasted a lot of my time. When I tried to post this as either a bug in the code or documentation, I had no expectation that someone would rush around and fix the code. I have done enough work on other open source projects. I was however pissed off at a refusal to spend five minutes adding a note to the documentation which would save other people wasting time like I did.

I have a few comments to add which may help to move forward:

• Please try to rolling back to the iousbfamily kext of 10.6.6 to 10.6.4. I have seen the MultiBeast? tool (see: ​http://tonymacx86.blogspot.com/), and you may also download the debugging versions of earlier usbfamily kext's (​http://developer.apple.com/hardwaredrivers/download/usbdebug.html). Not sure if it helps, but it may be worth a try.

Woah. Any body reading this: DON'T do that! You just cause the Mac to kernel panic. The kexts are matched (statically linked?) to the kernel. You cannot mix and match.

• northcob, please inquire with the Apple Developer Mailing list about how to write a codeless kernel extension for 10.6.6 that correctly seizes your device. After all, Apple provides tutorials for such codeless kext's, and they might be interested to provide updates to them so that these tutorials continue working after 10.6.6.

I think this idea that one can seize the device is misleading. The current codeless driver is loaded for the device, but since it just subclasses the AppleUSBComposite class without modification, it does not stop the driver loader querying the interfaces, getting a HID type, and loading the HID driver. What is needed is to override the method that reports the interface type. I am afraid that it is outside my skill set.

• the library structure of libusb-1.0 is capable to forcefully detach a kernel driver. This is currently only implemented for linux. Maybe either Nathan or someone else with good Darwin coding capabilities will be capable of implementing this feature also for Darwin.

I just don't think that approach can work. As long as the interface reports that it has the HID type (3) the HID driver will get loaded. I don't think it is possible for driver to get exclusive use of an instance of the AppleUSBComposite driver. I may e quite wrong about this because my understanding is sketchy at best.n If the above is right then the whole concept of usb_claim_interface may be impossible on Darwin. Or to put it another way, there may be a fundamental incompatability between the libusb model and the Darwin one.

• next a remark with usb versus hid: since any hid device is simply a usb device, I disagree with the comments above that libusb should not be able to handle hid devices. libusb can handle all hid devices and it correctly does so. Open it, initialize it, write to it, read from it, close it. This is how I access the OpenCockpits? hardware interfaces. It is pretty simple to handle HID devices as USB devices. No need for a HID abstraction layer.

It handles them on Linux. It fails on Darwin.

To repeat, I don't expect some magic solution in the near future but it is only fair to your users to leave it as a documented issue, not just bury it.

Writing stuff down is always good. I just had a bright idea that if we did a codeless kext that matched the interface instead of the device that might stop the HID driver loading. However, I am not convinced that usb_claim_interface will work even if the HID driver is not loaded because that new codeless kext would have the interface instead! It would not be free to claim. For the interface to be free, it probably has to fail matching completely or have type 0xFF.

I have to go now, but I might get to try it tomorrow.

Cheers
Bill

[northcob]

Replying to hjelmn:

As Xiofan says it is impossible to punt a kernel driver in OS X. To add this functionality it would require changes to IOUSBDevice and IOUSBLib. Its unlikely Apple would accept these changes for 10.5+ and impossible for anything older :(.

I expect there is absolutely no chance of this behaviour being changed. I guess that Apple have decided for security reasons that no user process, even with root privileges can change a driver. As far as I understand the docs, drivers are loaded by a kernel process which matches the parameters in the driver's plist with those reported by the device/interface. You just cannot interfere with that from user space.

Cheers
Bill

[ludovicrousseau]

Replying to northcob:

I was a bit pissed off because the essentially undocumented issues with libusb on Darwin wasted a lot of my time. When I tried to post this as either a bug in the code or documentation, I had no expectation that someone would rush around and fix the code. I have done enough work on other open source projects. I was however pissed off at a refusal to spend five minutes adding a note to the documentation which would save other people wasting time like I did.

northcob, the best you can do is to spend 5 minutes proposing a patch for the documentation. Since you will be the author of the documentation patch you will only have a full satisfaction.

Bye

[stuge]

Replying to ludovicrousseau:

northcob, the best you can do is to spend 5 minutes proposing a patch for the documentation.

Patches are always welcome. I've also added wiki edit permission to your user, northcob. Unfortunately it's off by default to spare the wiki from spam.

[northcob]

Replying to stuge:

Patches are always welcome. I've also added wiki edit permission to your user, northcob. Unfortunately it's off by default to spare the wiki from spam.

Kindly explain to me what is the point of a patch, if according to the powers that be in libusb there is no problem. There needs to be a problem before there is a fix!

Every time I try to post a problem it gets closed. Why should I have any expectation that they would be interested in a fix for a problem that they don't think exists?

Contrary to your message, it appears that contributions are not welcome.

Pointlessness reigns
Bill