The main thing that distinguishes systematic research from non-systematic is planning and thoroughness. An example of unplanned research is when you’re wandering page to page on Wikipedia. You think of a question that interests you or seems relevant, then a fact triggers another question which you then follow up with, etc.

An example of planned research is a meta-analysis. Meta-analyses usually state in advance the question they are trying to answer, where they’ll look for answers, what studies will qualify and which won’t, etc.

Another way to systematize research is to use spreadsheets to lay out all of the options and criteria you’ll consider in advance.

A few EA organizations rely on historical evidence to support the claim that what they are doing is high impact, but history is a great example of an area where it's easy to find support for or against any idea you’d like. For example, if you are making a case that protests have been successful in helping social movements succeed, it will be straightforward to find examples showing that is true.

However, there will also be many examples where protests have either done nothing or even hurt a movement.

A systematic way for the research to be done might be to compare the top 100 social movements on an objective metric of success. Then you could list all of the different strategies they used, then pull correlations between the top 10 most used strategies and the success level of the movement. That would dramatically change the landscape on how seriously one could take a claim that X strategy is better than Y strategy based on historical evidence.

Read the full article on systemic research by Joey at Effective Altruism Forum