Links to bug reporting techniques, tips and tools
Michael Horowitz is annoyed that users cannot easily tell if they have a major Adobe Reader security patch installed. Both the unpatched and patched versions report themselves as version 8.1.2. He reports various ways to check on different versions of Microsoft Windows, but even security software is having trouble checking correctly.
Upshot for vendors: a version number bump for patches is important!
Popularity: 45% [?]
Mark Shuttleworth describes the particular problems operating system distributions have with bugs: they are a collection point for bugs in many products and have a responsibility to their users to get the bugs to the places where they will be fixed:
Our primary goals should be to ensure that fixes we produce, and information we generate in the QA process, make their way upstream where they will benefit the broadest cross-section of the community. Separately, we want to ensure that each Ubuntu release ships without major issues, regardless of where those issues originated. We are responsible for the user experience of every line of code, even though we don’t produce every line of code.
For Shuttleworth, upstream cooperation is the key to providing the best user experience when it comes to distributing projects you did not write.
Popularity: 38% [?]
Dave Goldsmith has some experience trying to report a security vulnerability to a company that does not have a security-specific process:
I reply:
Can you give me some guidance on your response guidelines to security vulnerabilities? Is there a timeframe that you try and have vulnerabilities fixed by?
They reply:
Hi David,
We’re always looking for new ideas and fixes to roll out in future updates but as as rule we don’t comment on possibilities or timeframes.
His commenters discuss the lack of reporting guidelines for security flaws in websites. In a more recent post, he reveals that 37signals was the previously unnamed vendor, and writes:
Vulnerability reporting should not be handled in the same way that you manage feature requests.
[Via ZDNet, which discusses whether treating security flaws as defects is a more effective way to communicate the problems they cause.]
Issues to consider here are the usual with security bugs: vendor communication versus disclosure. In the case of security problems in end-user software the problem with disclosure is the race for users to patch their systems against attacks being launched on those systems, with consequences that vary per attacked system. When it’s a website with a security problem, disclosure has a different weight, as rather than leaving some users in the lurch while a fix is rushed out (or leaving them to fight for one), the vulnerability can largely be fully exploited immediately. There’s only one target. You may be writing off this vulnerability’s consequences all together in the hope of future responsiveness from the vendor.
Popularity: 64% [?]
The bMighty Blog describes how a bug in Firefox which was present in the 2.x series and was in the 3.0 released version has created some controversy: did the security researcher know about the bug earlier and choose the timing of the announcement purely for the publicity?
In other words, this is an issue that could have surfaced at any point in the past two or three years — but did not. Instead, news of the bug surfaced just five hours into biggest, and perhaps most important, software launch in Mozilla’s history.
The article goes on to describe various possible motivations for timing a security announcement with a highly publicised major release: it might benefit the researcher, but it also hurts Mozilla a lot. This is hostile bug reporting at its finest, if it wasn’t a coincidence.
Popularity: 51% [?]
Citect was notified of a buffer overflow bug in their remote plant management systems in January 2008 and has only just released a fix, writes TechNewsWorld:
“The problem is a classic example of buffer overflow from the ’90s,” Core Security CTO Ivan Arce told TechNewsWorld. “It’s not a very sophisticated thing, [which] makes it surprising.”
…
The flaw was first found in January, but Core Security says it was not corrected until just a few days ago.
“This could have been done better — especially on such a critical software,” Arce told TechNewsWorld. “It’s not somebody’s FTP server. It’s software that is critical and should be addressed in a more timely manner.”
Popularity: 39% [?]
In Kate Rhodes’s Best Practices for Web Developers, she addresses best practices for triage:
…lets be honest with ourselves. We’re actually ok with some things going “boom”. If we weren’t we’d be working for NASA. Every other development house I know of regularly releases software with bugs in it. As long as nothing too important breaks and nothing breaks in a way that leaves you looking like an idiot there’s a good chance you’re willing to live with it, for a while at lest. So, now that we’ve admitted the truth to ourselves, we can start triaging our app.
Popularity: 40% [?]
Marie Hagman writes a Bug Reporting Best Practices guide with a focus on reporting bugs that are likely to be fixed:
Bugs that less likely to be fixed:
It would be great to fix every bug in the product, but it’s also great to ship J. In prioritizing which issues to fix, here are the some factors that cause certain bugs to miss the triage bar. Bugs that are least likely to be addressed:
- Are not reproducible or very hard to reproduce. This may be because the problem occurs intermittently or there are not enough details in the bug report
- Have strange and complex steps to introduce failure
- Have no perceived customer impact
- Are edge cases
- Risk introducing greater instability through the bug fix then the bug itself causes. Especially late in the product cycle, the bar for these types of bugs is very high because the bugs that may be introduced by the fix we will likely not have time to fix.
Popularity: 55% [?]
The Register reports:
The quality of open source code has improved over the last two years, according to an audit sponsored by the US Department of Homeland Security.
The security and quality of more than 250 open source projects – including Apache, Linux, Firefox and PHP – was assessed using code analysis tools from Coverity as part of the federal government’s Open Source Hardening Project. Coverity set up a scan site that invited individual developers to put their code through its paces with its static source code analysis tool, Coverity Prevent.
Popularity: 40% [?]
Ubuntu is having a bug jam between 8th and 10th August 2008. As The Fridge says:
So, what is the Ubuntu Global Bug Jam? Put simply, it is a world-wide online and face-to-face event to get people together to fix Ubuntu bugs – we want to get as many people online fixing bugs, having a great time doing so, and putting their brick in the wall for free software. This is not only a great opportunity to really help Ubuntu, but to also get together with other Ubuntu fans to make a difference together, either via your LoCo team, your LUG, other free software group, or just getting people together in your house/apartment to fix bugs and have a great time.
Popularity: 54% [?]
ZDNet reports that the credit rating agency Moody’s incorrect rated a risky investment as having its top rating as an investment:
Computerworld UK quotes Ralph Silva, senior analyst at financial services advisory firm Tower Group, regarding rating agencies’ lackadaisical attitude toward technology management:
Ratings agencies never put sufficient emphasis on their technology resources,” he said. In spite of technology playing a key part in ratings decisions, “they simply haven’t felt getting technology right was important enough to business processes, unlike banks”.
Popularity: 41% [?]
Jonathan Lange reminds us via Focus Shift that the only measure of code quality is WTFs per minute:
Actually, this reminds me of something I heard a preacher say, “before I give a sermon, I go through it, find everything clever, and take it out” (I paraphrase, not having a reference on hand).
In as much as sermons and code should both be ego-free communications of ideas, I think this is sound advice for hackers.
Popularity: 44% [?]
The 2008 random number weakness in Debian (Ubuntu etc) versions of OpenSSL resulted in an article by Jake Edge for LWN: Debian, OpenSSL, and a lack of cooperation, covering the responsibilities of upstream and packagers:
It is in the best interests of everyone, distributions, projects, and users, for changes made downstream to make their way back upstream. In order for that to work, there must be a commitment by downstream entities—typically distributions, but sometimes users—to push their changes upstream. By the same token, projects must actively encourage that kind of activity by helping patch proposals and proposers along. First and foremost, of course, it must be absolutely clear where such communications should take place.
Popularity: 40% [?]
Joey Hess suggests that using the word “stupid” or using imperatives like “fix it” in bug reports is not a good way to get your bugs looked at. However Gunnar Wolf gives a recent example of an aggressive bug report that made him care.
Popularity: 82% [?]
Gustavo R. Montesino has released his Google Summer of Code project bug-triage, which according to the release announcement has these features:
- Added integration to upstream BTSes
- Added support to more advanced queries
- Added support to followup to a bug
- Allow selection of columns to show
- Added sorting support
- Allow to select the BTS access method to use (SOAP/BTS)
Popularity: 69% [?]
OpenQA is the home of several open source testing tools, notably Selenium, for automated testing of web applications in a number of browsers, including the major ones.
Popularity: 95% [?]
Sebastian Kuegler has a post on releasing KDE 4, and mentions briefly how the bug tracking tool could be used to clarify the release process:
What we can do to make it easier for vendors to ship the next version of our software is making it clear what needs to be done before a release. That would mean: “Here’s a list with what the next release will look like, if you want to make it happen earlier, help us hacking” This could be as easy as recording this in bugzilla, marking it as “showstopper” or “required for x.y.z” and then make the query showing those entries easily accessible through, for example, a direct link.
Popularity: 82% [?]
Luis Villa writes about Infotopia, information-gathering, and software QA, comparing QA problems in open source and proprietary software:
Quite simply, the big problem in QA is getting information about the state of the software out of the software and into the hands of developers as efficiently as possible.
This has three aspects: creating the information, getting it in the hands of the QA teams, and then filtering it into a form that is useful for developers to work on. Traditional QA has a very hard time getting the information… In contrast, open source QA has a whole ocean of information from the legions of volunteers willing to run pre-release code; the trick is to tap into that water without drowning in it.
Popularity: 74% [?]
One bugtracking system that has popped up since I was last posting here is Canonical’s Launchpad.
In mid-2006 James Henstridge talks about Launchpad in the context of it being considered as a bug tracker for Python (note that Python eventually chose Jira and then switched to Roundup). You can also check the more recent Feature Guide to Launchpad.
Popularity: 75% [?]
Jeff Waugh wants to report bad search results to Google, but where? His commenters have some good suggestions:
Scott Says:
June 17th, 2007 at 18:49Perhaps you have to be logged in to get this form?
["Dissatisfied with your search results?" form]
Willem Says:
June 17th, 2007 at 20:48Or bether yet (with some googling
)
Popularity: 95% [?]
Terence M. Colligan writes:
Although I thought I understood the importance of quality, and took pride in the quality of the software we produced, I never believed that delivering defect-free software was possible. After all, everyone knows that all software has lots of bugs, right?
Well, no, not necessarily! Certainly, most experiences with today’s software quality are not encouraging. Although few people can name even one piece of software which they use that has no bugs, defect-free software is possible to create. We know it is possible, because we’re doing it.
It started with a single engineer. This engineer was consistently producing work with a defect rate more than one hundred times smaller than our other engineers. She has done so for us for over three years now. During the same time, she has produced three to five times as much code as any other engineer.
I found this so exciting that I determined to find out how she did it, and to see if we could teach our other engineers to achieve the same quality results.
Via James Gregory, Defect-free code.
Popularity: 86% [?]
