Justin's Journal

Sometimes it is a lot easier to debug some of these hardware support issues in person as opposed to over IRC or bugzilla. If you have a laptop with hardware that isn't working quite right, and happen to be heading to flock, bring it with you.  I will be in the  Kernel regression and perf testing session to help debug some of these.  If you can't make that session, feel free to find me any time during the conference. If you don't have Fedora installed on these laptops, I will have USB keys with me to boot a live image for debugging purposes.

I know there has been a lot of press surrounding CVE-2016-8655 and particularly that the 4.8.13 stable kernel release did not include a fix.  While upstream stable did not include the fix until 4.8.14, Fedora did include the fix in our 4.8.13 releases. Those updates were in updates-testing for Fedora 23, 24, and 25 over the weekend, and are being pushed to the stable updates repository today. We encourage everyone to update, and not to wait for the 4.8.14 release in regards to this issue.

Bodhi updates:

Fedora 25
Fedora 24
Fedora 23

There are two main types of kernel bugs, those which fail loudly (oops!) and those which do not.  I am not counting new hardware support bugs, as those are really feature requests.  We have said many times recently, the Retrace Server is getting a lot of attention.  This is great, and quickly points out which of those noisy fails the most users are hitting on a given kernel release, and allows us to prioritize based on a combination of severity and the number of users hitting a given bug.  It can make things easier to track down, because we might have some idea of which kernel introduced this bug, and hopefully there is enough meat to the entry for us to know where to start looking for a solution.  Unfortunately, the second type of bug is not so clear.  This is things like regressions in hardware support (my wifi worked fine in the kernel x, but no longer works in kernel y), and various other regressions where silently, things no longer work as they should/used to.   What follows will hopefully make things easier in dealing with those second types of bugs.
There are three very important points that I should get out of the way first:

• Please test with the latest available kernel for the release you are running.  Any fix will be on top of that, so it saves a lot of time and effort on both parts to know that the bug still exists in the currently supported kernel.  When people file bugs against old versions, the first thing we will ask is if this is still occuring in the latest update.  Basically nothing will happen towards getting your bug fixed until we know that it does.


• Search bugzilla, and make sure no one else has filed a bug on this issue, if they have, don't open a new bug, comment on the existing bug.  Any new or relevant information can be a big help here. If you see something that looks similar, but not really what you are seeing, file a new bug. Adding that your wifi card doesn't work when docked on a bug about the same type of card not working at all will only cloud both issues, or get your issue ignored when the original is fixed and the bug is closed.


• Search google, and see if there has been discussion of this issue upstream or elsewhere.  If there has been, mention that in the bug description, link to that external information.  It saves us time, and can make a huge impact in how quickly your bug gets fixed.

As I am sure many of you are aware, bugzilla generates a lot of email. While the web interface does have some interesting search capability, email is the main method of getting notified of new bugs.   The better those initial emails (your bug reports) are worded, the more likely we can have a real understanding of the nature or priority of that bug. Now that retrace is decently usable, I tend to ignore any bugzilla mail from abrt and assume that it will show up in the retrace results if it is a high priority. That doesn't mean those bugs do not get addressed, only that we have a good system for prioritizing those outside of bugzilla.  This saves time and makes it easier to actually read through the new bugs filed against the kernel in other areas.  Bugzilla mail is sorted and works just like any other email list you might want to keep up with, only the "from" field is irrelevant, we can't see who filed the bug when scanning a mailbox.  The first thing we see of any interest is the subject.  Something generic here means I am less likely to take an interest after a long weekend when I have a lot of mail to go through, if your subject is clear and concise, this is more likely to grab attention.  Even better if it includes the first kernel version where this bug was seen.   Now that we have an idea of the actual issue, someone is more likely to take the time to read your description.  Again, clear and concise is good, but actual detail is important. If this is a matter of hardware no longer showing up between two kernel versions, tell us which was the last known good, and which was the first known bad. Mention that a dmesg from bootup with each of those kernels is attached (and attach them!). If behavior changed, explain exactly how it changed, and if you can show output from specific tools which will help, mention it in the description, and actually attach that output.  These types of details tell us that this is a bug where we have some idea of where to begin.   These are the types of details which get me to make sure to open your bug into a tab so I remember to go back to it when I finish scanning the new bug mail.  If this is a catastrophic failure, such as data corruption or the like, please mention that in both the subject and the description, it really helps us prioritize.  Finally, when we ask for more information, please try to respond.  Chances are, if we have asked for more information on your bug, it has been filed away as "done all I can do until I get a reply" which means I am unlikely to look at it again until I get an email saying there is new information.
Sorry this has been long winded, but it is something we on the kernel team spend a lot of time dealing with.  Anything we can do to streamline the process and correctly prioritize will be a big help towards a better working Fedora kernel, and that's good for everyone.

There are many reasons you might want to trigger events off of koji builds. I take kernel build completions as a trigger to automatically test the builds. I also use rawhide kernel build starts to kick off the scripts to build rawhide-nodebug kernels. Others might want to rebuild a dependent package, trigger some action off of a build failure, or push a git tree, etc. I thought I would share the snippet of code that actually monitors fedmsg for koji events in case it might be useful to others.

import logging
import fedmsg
import fedmsg.config
import fedmsg.meta

# Setup for the fedmsg listener
logging.basicConfig()
config = fedmsg.config.load_config([], None)
config['mute'] = True
config['timeout'] = 0
fedmsg.meta.make_processors(**config)

# Actually start the listener
for name, endpoint, topic, msg in fedmsg.tail_messages(**config):

    # Start looking for the items we care about
    if "buildsys.build.state.change" in topic and msg['msg']['instance'] == 'primary':
        matchedmsg = fedmsg.meta.msg2repr(msg, **config)
        if "completed" in matchedmsg and "kernel" in matchedmsg:
            #do what you want here

That is basically it. A few things to note:

• The Fedmsg "buildsys.build.state.change" topic relates to official koji builds, if you are looking for scratch builds, you would look for the "buildsys.task.state.change" topic.


• checking msg['msg']['instance']  will tell you which koji instance the message comes from. If you don't do this, it will come from all instances currently reporting (primary, arm, ppc) this can mean repeated triggers as most packages built on primary are later built on the other instances,


• matchedmsg is essentially the "human readable message at this point and easy to search. an example message looks like: buildsys.build.state.change -- labbott's kernel-4.3.0-0.rc5.git1.1.fc24 completed http://koji.fedoraproject.org/koji/buildinfo?buildID=691944


• This isn't the absolute most efficient method here, the msg['msg'] dict contains a good bit of interesting information.  For instance, I could check for completed before running fedmsg.meta.msg2repr on it. This would eliminate more than 50% of those calls, but performance hasn't been an issue. For the ease of blogging this, it is easier to tell you to search for the desired build state by name than tell you that msg['msg']['new'] == '4' means that the build state is now complete.

Fedmsg is also quite useful for several other things in the Fedora infrastructure, and it is very well documented.

Several of you have been running the kernel test suite. We appreciate it! I know it has been a bit of a PITA to get us results by manually submitting them on the website. Today, that has changed. We support automatic submissions now. To enable this, you can copy the config.example file in the git checkout to .config, edit as necessary and your logs will be submitted in the method that you choose. The .config file is fairly simple, and it lets you choose whether to submit anonymously, FAS authenticated in order to get badges, or not at all, and everything will continue to work as it has been.

We appreciate those of you who have taken time to run the test suite on kernel builds. Hopefully this will make the process a bit easier.

We have been working on the kernel test infrastructure for a while now, but we didn't have anywhere for people to actually see the results. Now the front end is live on apps.fedoraproject.org/kerneltest. Not only can you see the results, you can upload your logs, and earn badges for doing so! Right now, running the tests take a little bit more effort on your part, and that is the next thing to be simplified. Here is how things work now:

git clone https://git.fedorahosted.org/git/kernel-tests.git
cd kernel-tests/
sudo ./runtests.sh

When the test suite finishes, it will tell you where the log file is, and point you to the front end site to upload them.
In the future, we will have an opt-in to auto submit logs, and an actual package in Fedora vs having to deal with git directly.
Submitting a test can earn you this awesome badge. Soon there will be additional badges for continued effort in helping us test the kernel.

Many of you have been using the Rawhide Nodebug repository to test the 3.12 stable kernels while we got Fedora 20 out the door. We really appreciate the feedback from those who have tested. This is a heads up that in one week (December 25th) the rawhide nodebug repository will return to tracking rawhide and the 3.13 development cycle. Both Fedora 19 and Fedora 20 have the most recent 3.12.5 kernel submitted for updates-testing. If you wish to remain on rawhide nodebug and test development kernels, we really appreciate it. If you wish to remain on a 3.12 kernel, now is the time to disable the rawhide-nodebug repository. To do so, simply flip the value for enabled from 1 to 0 in /etc/yum.repos.d/fedora-rawhide-kernel-nodebug.repo.

We have a slight issue with the 3.12 kernel timing in that it is too late to push it into Fedora 20, but too far away from the Fedora 20 release to just ignore the 3.13 development cycle until release. As a result, we will be tracking 3.12 and stable updates for it in the rawhide-nodebug repository. This gives us a chance to keep it built and tested on all primary architectures, and make sure we are in good shape to push 3.12 out as an update as soon as possible. Once 3.12 can be pushed to releases, the rawhide-nodebug repository will return to doing non debug builds of rawhide, tracking Linus' tree upstream. I will let everyone know that is happening through the same channels with a couple of days notice.