OpenBSD: socket.recvmsg tests fail with OSError: [Errno 40] Message too long

[vstinner]

BPO	20669
Nosy	@jcea, @ncoghlan, @pitrou, @vstinner, @jackdied, @giampaolo, @cedk, @brianmay, @worr
Files	recvmsg_trunc_emsgsize.patch recvmsg_linux_freebsd_only.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-07-03.14:29:10.398>
created_at = <Date 2014-02-18.08:41:31.292>
labels = []
title = 'OpenBSD: socket.recvmsg tests fail with OSError: [Errno 40] Message too long'
updated_at = <Date 2017-07-03.14:29:10.396>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-07-03.14:29:10.396>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-07-03.14:29:10.398>
closer = 'vstinner'
components = []
creation = <Date 2014-02-18.08:41:31.292>
creator = 'vstinner'
dependencies = []
files = ['34158', '39096']
hgrepos = []
issue_num = 20669
keywords = ['patch']
message_count = 15.0
messages = ['211493', '211757', '211762', '211767', '211827', '211834', '211967', '240918', '241073', '241092', '241243', '241393', '297120', '297121', '297597']
nosy_count = 18.0
nosy_names = ['jcea', 'ncoghlan', 'janssen', 'pitrou', 'therve', 'vstinner', 'jackdied', 'baikie', 'giampaolo.rodola', 'ced', 'synapse', 'wiml', 'neologix', 'rosslagerwall', 'python-dev', 'brian', 'rpointel', 'worr']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue20669'
versions = ['Python 3.5']

[vstinner]

http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x/builds/14/steps/test/logs/stdio

======================================================================
ERROR: testCmsgTrunc1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2857, in testCmsgTrunc1
    self.checkTruncatedHeader(self.doRecvmsg(self.serv_sock, len(MSG), 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTrunc2Int (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2867, in testCmsgTrunc2Int
    SIZEOF_INT * 2))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2907, in testCmsgTruncLen0
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0), maxdata=0)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Minus1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2874, in testCmsgTruncLen0Minus1
    socket.CMSG_LEN(0) - 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Plus1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2913, in testCmsgTruncLen0Plus1
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0) + 1, maxdata=1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2920, in testCmsgTruncLen1
    maxdata=SIZEOF_INT)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen2Minus1 (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2927, in testCmsgTruncLen2Minus1
    maxdata=(2 * SIZEOF_INT) - 1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1872, in doRecvmsg
    result = sock.recvmsg(bufsize, *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testFDPassSeparate (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2722, in _testFDPassSeparate
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
ERROR: testFDPassSeparateMinSpace (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2746, in _testFDPassSeparateMinSpace
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
ERROR: testCmsgTrunc1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2857, in testCmsgTrunc1
    self.checkTruncatedHeader(self.doRecvmsg(self.serv_sock, len(MSG), 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTrunc2Int (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2867, in testCmsgTrunc2Int
    SIZEOF_INT * 2))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2907, in testCmsgTruncLen0
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0), maxdata=0)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Minus1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2874, in testCmsgTruncLen0Minus1
    socket.CMSG_LEN(0) - 1))
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen0Plus1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2913, in testCmsgTruncLen0Plus1
    self.checkTruncatedArray(ancbuf=socket.CMSG_LEN(0) + 1, maxdata=1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2920, in testCmsgTruncLen1
    maxdata=SIZEOF_INT)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testCmsgTruncLen2Minus1 (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2927, in testCmsgTruncLen2Minus1
    maxdata=(2 * SIZEOF_INT) - 1)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2888, in checkTruncatedArray
    len(MSG), ancbuf)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1963, in doRecvmsg
    result = sock.recvmsg_into([buf], *args)
OSError: [Errno 40] Message too long

======================================================================
ERROR: testFDPassSeparate (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2722, in _testFDPassSeparate
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
ERROR: testFDPassSeparateMinSpace (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 258, in _tearDown
    raise exc
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 270, in clientRun
    test_func()
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2746, in _testFDPassSeparateMinSpace
    array.array("i", [fd1]))]),
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 1864, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument

======================================================================
FAIL: testFDPassSeparate (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2710, in testFDPassSeparate
    maxcmsgs=2)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

======================================================================
FAIL: testFDPassSeparateMinSpace (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2734, in testFDPassSeparateMinSpace
    maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

======================================================================
FAIL: testFDPassSeparate (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2710, in testFDPassSeparate
    maxcmsgs=2)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

======================================================================
FAIL: testFDPassSeparateMinSpace (test.test_socket.RecvmsgIntoSCMRightsStreamTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2734, in testFDPassSeparateMinSpace
    maxcmsgs=2, ignoreflags=socket.MSG_CTRUNC)
  File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_socket.py", line 2637, in checkRecvmsgFDs
    self.assertEqual(msg, MSG)
AssertionError: b'' != b'Michael Gilfix was here\xe1\x88\xb4\r\n'

[vstinner]

Extract of the POSIX standard for recvmsg():
http://pubs.opengroup.org/onlinepubs/009695399/functions/recvmsg.html

"""
DESCRIPTION

(...) The recvmsg() function shall return the total length of the message. (...) If a message is too long to fit in the supplied buffers, and MSG_PEEK is not set in the flags argument, the excess bytes shall be discarded, and MSG_TRUNC shall be set in the msg_flags member of the msghdr structure. (...)

ERRORS

[EMSGSIZE] The msg_iovlen member of the msghdr structure pointed to by message is less than or equal to 0, or is greater than {IOV_MAX}.
"""

I don't see EMSGSIZE in the Linux manual page of recvmsg(). So it looks like Linux truncates, whereas OpenBSD raises an error. It's probably safe to accept in the unit test that the OS can raise OSError(EMSGSIZE).

[vstinner]

recvmsg_trunc_emsgsize.patch: modify recvmsg() tests checking for truncated data to ignore the test on OSError(EMSGSIZE).

[vstinner]

For failing "FD pass" tests, see issue bpo-12958: these tests are also skipped on Mac OS X. We should maybe also skip these tests on OpenBSD.

[vstinner]

I copy the nosy list from issue bpo-6560 which added the recvmsg() and sendmsg() methods.

[vstinner]

I opened the issue bpo-20718 to track the "FD pass" failures.

[baikie]

I opened the issue bpo-20718 to track the "FD pass" failures.

But the failures in the current issue *all* involve FD passing :)

The "Message too long" errors are in tests where the ancillary data (in this case file descriptors) is truncated, rather than the normal data, so msg_iovlen shouldn't be a problem there (and MSG_CTRUNC would normally be set, rather than MSG_TRUNC).

I suppose the most likely reason for a process to be sent too many file descriptors is that someone is attempting a denial-of-service attack on it, so perhaps returning EMSGSIZE is intended as a "fail safe" response, even if it isn't blessed by POSIX?

[worr]

In the case of EINVAL, OpenBSD returns this if multiple cmsghdrs are attached to a message with sendmsg(2). In this case, those tests should probably be skipped.

I've attached the patch that combines haypo's patch with skips referencing this issue wrt. multiple cmsg failures

[worr]

I'm adding another patch, as adding multiple cmsg's to a single message with sendmsg is a Linux-specific extension. This only enables the problematic patches on Linux. This solves the AIX failures in bpo-20718 as well.

[vstinner]

Can you please split the changes to skip tests on non-Linux platforms, and changes for OpenBSD/AIX please?

[baikie]

Sending multiple control messages at once isn't Linux-specific - the tests in question passed on FreeBSD.

[worr]

Double checked on a FreeBSD box and confirmed. I missed the loop in the kernel source.

I've attached a new patch, without the emsgsize change.

[vstinner]

Oh, this issue is now old. Can someone please try the patch? Does it fix the issue?

[vstinner]

Or should we just close the issue as "out of date"?

[vstinner]

I'm sorry but I don't have OpenBSD, and there is no more OpenBSD buildbot, so I close this issue as out of date :-( We failed to find anyone to review the patch since 2 years.