Software bugs

Dave Goldsmith has some experience trying to report a security vulnerability to a company that does not have a security-specific process:

I reply:

Can you give me some guidance on your response guidelines to security vulnerabilities? Is there a timeframe that you try and have vulnerabilities fixed by?

They reply:

Hi David,

We’re always looking for new ideas and fixes to roll out in future updates but as as rule we don’t comment on possibilities or timeframes.

His commenters discuss the lack of reporting guidelines for security flaws in websites. In a more recent post, he reveals that 37signals was the previously unnamed vendor, and writes:

Vulnerability reporting should not be handled in the same way that you manage feature requests.

[Via ZDNet, which discusses whether treating security flaws as defects is a more effective way to communicate the problems they cause.]

Issues to consider here are the usual with security bugs: vendor communication versus disclosure. In the case of security problems in end-user software the problem with disclosure is the race for users to patch their systems against attacks being launched on those systems, with consequences that vary per attacked system. When it’s a website with a security problem, disclosure has a different weight, as rather than leaving some users in the lurch while a fix is rushed out (or leaving them to fight for one), the vulnerability can largely be fully exploited immediately. There’s only one target. You may be writing off this vulnerability’s consequences all together in the hope of future responsiveness from the vendor.

Popularity: 64% [?]

The bMighty Blog describes how a bug in Firefox which was present in the 2.x series and was in the 3.0 released version has created some controversy: did the security researcher know about the bug earlier and choose the timing of the announcement purely for the publicity?

In other words, this is an issue that could have surfaced at any point in the past two or three years — but did not. Instead, news of the bug surfaced just five hours into biggest, and perhaps most important, software launch in Mozilla’s history.

The article goes on to describe various possible motivations for timing a security announcement with a highly publicised major release: it might benefit the researcher, but it also hurts Mozilla a lot. This is hostile bug reporting at its finest, if it wasn’t a coincidence.

Popularity: 51% [?]

Marie Hagman writes a Bug Reporting Best Practices guide with a focus on reporting bugs that are likely to be fixed:

Bugs that less likely to be fixed:

It would be great to fix every bug in the product, but it’s also great to ship J. In prioritizing which issues to fix, here are the some factors that cause certain bugs to miss the triage bar. Bugs that are least likely to be addressed:

• Are not reproducible or very hard to reproduce. This may be because the problem occurs intermittently or there are not enough details in the bug report
• Have strange and complex steps to introduce failure
• Have no perceived customer impact
• Are edge cases
• Risk introducing greater instability through the bug fix then the bug itself causes. Especially late in the product cycle, the bar for these types of bugs is very high because the bugs that may be introduced by the fix we will likely not have time to fix.

Popularity: 55% [?]

Joey Hess suggests that using the word “stupid” or using imperatives like “fix it” in bug reports is not a good way to get your bugs looked at. However Gunnar Wolf gives a recent example of an aggressive bug report that made him care.

Popularity: 82% [?]

Jeff Waugh wants to report bad search results to Google, but where? His commenters have some good suggestions:

Scott Says:
June 17th, 2007 at 18:49

Perhaps you have to be logged in to get this form?

["Dissatisfied with your search results?" form]

Willem Says:
June 17th, 2007 at 20:48

Or bether yet (with some googling )

["Inappropriate or irrelevant search results" form]

Popularity: 95% [?]

Jacub Steiner has some tips for bug reporters who are making a feature request (ie asking for new functionality, rather than reporting broken existing functionality):

If you take the time to write a functional specification, you are very likely going to motivate someone to get it implemented. You take the burden of designing the behaviour and let the developer worry about implementation details, data structures, etc. You invest your time and effort just like the hacker would have. Here’s a few suggestions.

• Define the functionality by creating a few test cases on what problems you are trying to solve.
• Be very specific and go through the process step by step. It will make you find problems with your design. If you were to just suggest functionality and don’t work out the details, you will, in the best scenario, waste developer’s time discussing the flaws you would have found yourself.
• It is better to find a bad design while writing the spec than after it has been implemented. Convincing a developer to redo something is close to impossible when he already invested a lot of time in it.
• Be visual. Create mockups of the interface as you go step by step through the process of solving the task. Many times I have thought my descriptions are clear, when they weren’t. Images tend to be less vague (Well this may simply be because my writting sucks, too).

Popularity: 67% [?]

Havoc Pennington describes the coordinated release process for security bugs:

The simplest thing is to quietly notify any of the major Linux or BSD distributions and let them take it from there… Once you notify someone, wait to hear back. The upstream maintainer would normally announce the vulnerability and commit patches to CVS at the same coordinated time that vendors post packages. If you patch in CVS before anyone is ready with packages, your users are vulnerable during the gap (and generally unhappy about it). Worse, by committing a patch to CVS you’re doing something that a black hat could notice, but most sysadmins will not notice.

Popularity: 100% [?]

Reporting a program’s vulnerability to an attack of some kind is a bit different to reporting normal bugs, because public exposure of the vulnerability may cause it to be exploited before the developers can develop a fix. Hence, normal bug reporting procedures, such as making an entry in a project’s public bug database, may not be the best approach. In this case, you may want to approach the vendors or developers privately, even if it goes against their bug reporting guidelines.

CERT’s policy (via SLUG) gives vendors at least 45 days to address a security issue before advising the public. Their Vulnerabilities, Incidents, & Fixes FAQ gives some guidelines for people who have discovered vulnerabilities.

The Organisation for Internet Safety has also issued a document called "Guidelines for Security Vulnerability Reporting and Response Process – V1.0" [780kb PDF file].

Popularity: 58% [?]

The Eclipse project has a set of bug writing guidelines, including a great deal of detail about what should go in bug reports. It also gives some motivation for writing good bug reports.

Popularity: 44% [?]

Eric S Raymond and Rick Moen have a HOWTO on asking questions of technical groups the smart way. See in particular the section about not assuming a particular behaviour is a bug.

Popularity: 46% [?]

Good technical questions have many similarities with good bug reports: see in particular The Cardinal Rule of Reporting Technical Problems:

Never, never, never, never, never say `doesn’t work’.

Never.

Proper Prophylaxis:

Just say “I wanted X, but it does Y. How do I get X?”

Popularity: 52% [?]

Joel Spolsky illustrates painless bug tracking by example, advocating using a bug database and advising how to make people use it. He has a second article on the need for a good QA team.

Popularity: 59% [?]

The KDE Community Wiki site has a guide to getting involved by helping out with bug triaging. It also contains more useful tips regarding making a good bug report.

Popularity: 63% [?]

php.net has some tips (some are Bugzilla specific) for handling bug reports at the bug-fixer end.

Popularity: 38% [?]

Telsa sent a link to her Making the right sort of difference: bugs and what to do with them summary from her talk at linux.conf.au 2003.

She writes that she has recently had an opportunity to confirm that bug numbers are increased by an approaching deadline…

It covered a lot of material: finding bugs, reporting bugs and getting people to fix bugs for you.

You can also find an Ogg Speex audio recording of the talk at planetmirror (Australia only) or on the LCA 2003 ISO image distributed by various mirrors (see the linux.conf.au 2003 site).

Popularity: 58% [?]

… or why your browser bug didn’t get fixed.

Via Malcolm again, Dave Hyatt describes ways not to get your bugs fixed and the sheer number of potential browser bugs.

Popularity: 51% [?]

Via the Mozilla QA team’s page: bug writing guidelines for Mozilla and Bugzilla etiquette.

Popularity: 35% [?]

The GNOME Bugsquad are the team who track current GNOME bugs.

They also have several resources for bug hunters including their own Bugzilla guide and a guide to triaging bugs.

Popularity: 60% [?]

The last time this site was active (2000), Eli Goldberg sent me a link to bug writing guidelines. The guide has some general tips and also some specific tips for reporting bugs with Bugzilla.

He also sent a guide to text searching in Bugzilla and to bug triaging in Bugzilla.

Popularity: 35% [?]

Simon Tatham, the author of PuTTY, has an article on how to report bugs effectively.

Popularity: 38% [?]