[Click here for the previous post in this series]
“It doesn’t matter how beautiful your theory is. If it disagrees with experiment, it’s wrong. In that simple statement is the key to science.” -Richard Feynman
While much of this series has been about writing an exciting story, we now need to put excitement aside for a while. I’ve earlier claimed that papers are not only containers of information. Their Methods sections, however, are. Their role is entirely utilitarian. So before we discuss form, let’s discuss function.
A Methods section serves two purposes. First, it should let other researchers gauge whether your conclusions are justified and backed up by evidence—it should let other researchers assess how credible your data are, and how credible your analysis is. Second, it should allow other researchers to replicate your study and repeat whatever it is that you have done.
Unfortunately, as any experienced researcher knows, these goals are not always met. More often than not, the authors of a paper do not explain the procedures that they have used in enough detail, even if there is a Supporting Information document with an unrestricted page count. It happens all too often that when the reader attempts to understand in detail how the authors have arrived at their results, she has to give up because that information is simply not there or it is too patchy.
Not being able to understand a paper’s methods or to replicate its pipeline leads to many problems. First, this contributes to the replicability crisis and therefore erodes the very foundation of science, the scientific principle itself: only those results that can be replicated by others can be taken as facts. Second, selling your discovery to the scientific community will be hard if your fellow scientists cannot trust your findings because they do not understand how they were obtained. Third, if your pipeline—from data collection to analysis—contains new methods or ideas, those will not be adopted by anyone unless they are clearly explained (or even wrapped up and served on a plate, say, as a software package). This leads to many lost citations and your work not being discovered. If you release data, someone can also use it for things that you didn’t think of, and if you release software, there will always be someone who needs it.
So please do take replicability and reuse seriously. Explain what you have done in as much detail as possible. Release your raw data. Release your intermediate results. Release your code. Reveal everything. Hide nothing. Be a good scientist. Don’t be an evil scientist.
If you release everything that there is to release, you will probably need to use external repositories. Some journals, however, do allow submitting supplementary data and code files, to be published together with the article. If you are thinking of hosting the data and code yourself, consider that we are talking about the scientific record here: your paper, your data, and your code should, in theory, be available forever. And forever is a mighty long time, as the late artist known as Prince once put it. It certainly is longer than the lifetime of the URL that points to your www homepage on your university’s server, or of the server daemon that runs on the Linux machine in your bedroom closet. So no DIY here, please—always use long-term data and code repositories, like Zenodo. While even those might not last forever, they’ll last longer than any self-hosted repository. Note that even GitHub is not futureproofed: it is run by a commercial company that can become extinct just like any other company.
Let’s return to the paper itself, and move from function to form. First, where to describe materials, data, and methods? This, of course, depends on the journal, and there are many options. The top-tier journal style (think PNAS, Nature, etc) is to have Materials and Methods as a separate section at the end of the article, as an appendix of sorts. In these journals, methods are only briefly described in the main text and the reader is referred to the Materials and Methods appendix for details. While writing a paper this way may at first feel difficult, this structure does make sense: the short letter format is all about the story, and technical details that would get in the way of the story are pushed aside. This may make writing feel harder because one cannot hide behind technical details: there has to be a story. However, beware of the dark side: referring the reader to the Materials and Methods section where only superficial details are given and where the reader is further referred to the Supplementary Information that adds detail but still lacks essential information, or where the limitations of the chosen methods are hidden in a subordinate clause on page 28. This structure makes it dangerously easy to sweep something under the rug. Which is why it often happens.
So if you are writing for one of these journals, do resist the dark side: do not hide problems in the SI. Other than that, just strive for clarity in the Materials and Methods. Typically this section comprises independent subsections for different items, so there is not much storytelling involved. In the main text, when talking about methods, describe their purpose, not their details: “we measure the similarity of X and Y with the help of (insert name of fancy similarity measure), see Materials and Methods for details.”
Then there is the style common to biomedical journals, where Methods are described in all their detail straight after the Introduction. This makes it easier to describe everything properly and more difficult to hide problems, which is good. The downside is that being hit by several pages of painstakingly detailed method descriptions is something of a turn-off: the story suffers. While this cannot be entirely avoided, it helps if you remember to provide context: begin each subsection by reminding the reader why this data set was collected, why this experiment was done, or why you are going to next describe some mathematical methods. Often, this is not more difficult than simply saying that, e.g., “to measure the similarity of X with Y, we need some well-behaved distance measure for probability distributions that…” and then describing the chosen measure.
The third way that is common, say, to the journals of the American Physical Society, is to happily mix methods with results, explaining how things were done and what the outcome was without making a distinction between the two. In this case, things like experimental setups or data collection procedures may still be explained separately, but typically all mathematical and statistical methods are described together with the results. In my view, this makes writing a smoothly flowing story easier than the biomedical style. It is easier to motivate the methods by saying that “next, we’ll investigate X, and to do that, we need to do Y, and look, here’s the result”. In the biomedical style, this connection is harder to make because the methods and results are separated, so one has to focus on making sure that the reader understands why the methods have been chosen and why the reader should understand their details.
Before concluding, let us return back to being good versus being evil, and talk about discussing the limitations of your methods. All methods have limitations, as every scientist knows, and it is best to lay these out in the open. In my view, the Methods section is the best place for doing this: while even minor limitations of methods are often discussed in the Discussion section, it feels more natural if they are addressed when the methods are introduced. Strangely, this even feels more honest. First, at least to me, it feels a bit like having been cheated if I have read a long paper, and only in the last paragraph, it is mentioned that by the way, we’re not sure that things work the way we just told they would. Second, it is easier for the writer to explain the limitations together with the methods. Third, it is also easier for the reader to understand the limitations and their implications if the details of the methods are fresh in her memory.
When addressing limitations, you should tread carefully: being honest is different from making it sound like your study is flawed. Joshua Schimel’s “Writing Science” introduces a great principle: say but, yes instead of yes, but. Instead of saying that your quite clear results would be much more detailed if your experimental setup would have a higher resolution (or similar), say that even though the resolution of your experimental setup is limited, your results are quite clear. The latter has a much more positive ring to it, although both sentences have the same information content. So don’t make it sound like there is something wrong with your work—if there is, fix it first, before writing your paper.
Coming up next: the Results section.