#2312 closed defect (fixed)

DeprecationWarning about passing non-bytes header values

Reported by: daira Owned by: warner
Priority: minor Milestone: 1.10.1
Component: code Version: 1.10.0
Keywords: DeprecationWarning cygwin tests Cc:
Launchpad Bug:



DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead.

I don't know why this is warning only on the "Marcus Cygwin WinXP" builder.

Change History (16)

comment:1 Changed at 2014-09-30T19:41:17Z by daira

Ah, it is also happening on the Python 2.6 Travis build: https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/36701619

The common factor is Python 2.6.

comment:2 Changed at 2014-10-02T01:03:42Z by daira

  • Keywords easy added
  • Milestone changed from undecided to 1.11.0
  • Owner set to daira
  • Status changed from new to assigned

comment:3 Changed at 2015-01-20T17:49:16Z by warner

  • Owner changed from daira to warner
  • Priority changed from normal to minor
  • Status changed from assigned to new

This is low-priority, but it'd be nice to fix. It appears that something is calling http.Request.write with a unicode string, but we need a larger stack trace to figure out what's happening.

comment:4 Changed at 2015-01-21T00:46:50Z by warner

A minimal STR is to test test_web.Web.test_GET_DIRURL_json under python2.6 .

It turns out that the twisted.web code is expecting the Content-Length header to contain bytes (like all other headers, who knew?). Three Tahoe code paths (in web/common.py and web/filenode.py) fill it with an integer, which takes advantage of a ("%s" % value) fallback.

A DeprecationWarning to complain about non-bytes was added in Twisted-12.3, probably to guard against unicode header values, but in this case it's triggering on our IntType. Twisted-12.3 is also when twisted.python.compat.intToBytes(n) was added to make this easier, but it isn't present in 11.0-12.2, which is what we use on windows. So we'll need our own intToBytes function.

comment:5 Changed at 2015-01-21T01:48:27Z by warner

Oh, and category=DeprecationWarning is globally disabled in python2.7, because(?) they assumed it would only be used for features implemented by python itself, and "2.7 is the last one". I think that's why this doesn't appear in py2.7 .

Sigh, that seems like a dangerous misfeature to me. Should Twisted (and other projects) use a subclass or alternative to DeprecationWarning?

comment:6 Changed at 2015-01-21T08:48:28Z by warner

After fixing that, I found one other instance, when various web requests use nevow.rend.Page's feature of self.addSlash==True. This code path causes setHeader("location", url) to be called with a nevow.url.URL instance, instead of a string. There's no good way for us to fix this from tahoe.. the fix really needs to be in Nevow itself. Filed as https://github.com/twisted/nevow/issues/52 .

comment:7 Changed at 2015-01-21T18:33:42Z by warner

  • Keywords needs-review added

Ok, PR in https://github.com/tahoe-lafs/tahoe-lafs/pull/137 , ready for review. This will fix the immediate problem, but there will probably be others (including ones we can't fix because they're in Nevow or other libraries). I'll work on setting up a buildslave that will run tests with PYTHONWARNINGS=default::DeprecationWarning so we can at least see where they're happening.

comment:8 Changed at 2015-01-21T18:44:23Z by warner

#2366 created to remind me about the warnings-scanning buildbot tool

comment:10 Changed at 2015-01-22T17:29:06Z by zooko

  • Keywords needs-review removed

removing review-needed tag (which by the way was incorrectly spelled needs-review)

The next step is: figure out why the patch didn't fix the issue?

comment:11 Changed at 2015-01-22T21:25:53Z by daira

  • Keywords easy removed

comment:12 Changed at 2015-01-27T08:38:25Z by warner

I suspect the remaining warnings are resulting from Nevow's addSlash=True problem. I don't think we can fix that within Tahoe, but I could be wrong.

I think we've gotten as much out of this ticket as we can for 1.10.1 . I propose we bump it to 1.11 at this week's triage.

comment:13 Changed at 2015-01-27T18:39:02Z by daira

If we can't fix it, should we suppress the warning? We have done so for previous warnings that could only be fixed in dependencies.

comment:14 Changed at 2015-03-10T16:36:35Z by warner

let's make sure that this has been reported upstream to Nevow, if possible

comment:15 Changed at 2015-03-20T08:34:19Z by warner

Nevow bug filed: https://github.com/twisted/nevow/issues/52 . (oh, as I reported earlier in comment:6, nevermind).

Buildbot is now scanning for deprecation warnings, https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Ubuntu%20trusty%2014.04/builds/45/steps/deprecations/logs/warnings has a list. This is a simple regexp scan, so some of the lines are coming from the tests that fail (due to stderr not being as quiet as it's supposed to be):

/home/bb-tahoe/bb-tahoe/Ubuntu_trusty_14_04/build/support/lib/python2.7/site-packages/Nevow-0.11.1-py2.7.egg/nevow/appserver.py:129: DeprecationWarning: twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0: Please use twisted.web.http.Request.requestHeaders instead.
/usr/lib/python2.7/dist-packages/twisted/web/server.py:217: DeprecationWarning: Passing non-bytes header values is deprecated since Twisted 12.3. Pass only bytes instead.
a = '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n  import sets as _sets\n'
twisted.trial.unittest.FailTest: ('allmydata-tahoe: 1.10.0.post287 [master: 681eb4d12e20d8818ee02baab7b55230a706e76d]\nfoolscap: 0.6.4\npycryptopp:\nzfec: 1.4.5\nTwisted: 13.2.0\nNevow: 0.11.1\nzope.interface: unknown\npython: 2.7.6\nplatform: Linux-Ubuntu_14.04-x86_64-32bit_ELF\npyOpenSSL: 0.13\nsimplejson: 3.3.1\npycrypto: 2.6.1\npyasn1: 0.1.7\nmock: 1.0.1\nsetuptools: 0.6c16dev5\nservice-identity: 14.0.0\ncharacteristic: 14.3.0\npyasn1-modules: 0.0.5\n\n', '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated\n  import sets as _sets\n', 0)
twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/usr/lib/python2.7/dist-packages/twisted/spread/jelly.py:92: DeprecationWarning: the sets module is deprecated

So there are three real warnings:

  • nevow/appserver.py:129: "twisted.web.http.Request.received_headers was deprecated in Twisted 13.2.0"
  • twisted/web/server.py:217: "Passing non-bytes header values is deprecated since Twisted 12.3"
  • twisted/spread/jelly.py:92: "the sets module is deprecated; import sets as _sets"

The first is filed as https://github.com/twisted/nevow/issues/53 , and the second as https://github.com/twisted/nevow/issues/52 . I don't know if the third is filed anywhere.

comment:16 Changed at 2015-03-26T01:56:41Z by warner

  • Resolution set to fixed
  • Status changed from new to closed

All the bugs (that might ever get fixed) are filed, a separate ticket will track the creation of a buildbot to look for more. I think this ticket is done.

Note: See TracTickets for help on using tickets.