RESEARCH PRINCIPLES

What can be trusted, and how it is checked

The principles for checking and evaluating the results of AI research, and the principles that the operation of the Scholars’ Council for AI itself should follow.

Operating principles

What principles should the Scholars’ Council for AI itself follow?

The operation of the Scholars’ Council for AI itself should follow the same principles.

The roles of those who commission the research, those who carry it out and those who review it as scholars need to be made clear, and interests that could affect the results need to be recorded; even when research results differ from expectations, it should be possible to review them. Scholarly independence is given concrete form through these procedures and records.
The public-interest purpose of the Scholars’ Council for AI is also reflected in the selection of projects and the review of results. When the requests of the funding organization differ from the needs of the people actually affected, that difference is recorded and reviewed.

Read more — The operating principles the Scholars’ Council for AI seeks to keep

The operation of the Scholars’ Council for AI itself should also follow the same principles. The roles of those who commission the research, those who carry it out and those who review it as scholars need to be made clear, and interests that could affect the results need to be recorded. Even when research results differ from expectations, it should be possible to review them. Scholarly independence is given concrete form through these procedures and records. The public-interest purpose of the Scholars’ Council for AI is also reflected in the selection of projects and the review of results. When the requests of the funding organization differ from the needs of the people actually affected, that difference is recorded and reviewed. Participants in the field should have the opportunity to explain difficulties in use and unexpected results. The scholarly judgment of the faculty and experience in the field are reviewed together, while it is made clear who makes the final operating decisions and who takes on the follow-up measures.

Trust and evidence

How far can results be trusted?

The ability to extract facts accurately from documents, the ability to find rules that conflict with one another and the ability to act within set authority are different objects of evaluation. If only a single overall score is given, it is hard to know in which situations a system can be used. Other researchers can review the results only if the task inputs and model version, the state of the tools and materials, and the settings and evaluation criteria are preserved.

Read more — Checking whether results can be trusted

To discuss the trustworthiness of AI, it should be made specific what it is that is being called trustworthy. The ability to extract facts accurately from documents, the ability to find rules that conflict with one another and the ability to act within set authority are different objects of evaluation. If only a single overall score is presented, it is hard to understand in which situations a system can be used. Measurement validity is a question of whether an indicator appropriately reflects the intended concept. If only the number of messages is counted when evaluating collaboration in an AI team, it may not be possible to distinguish a team with lively dialogue from a team with a great deal of unnecessary repetition. The fact that the evaluator is a model is also subject to review. The AI that generated a result and the AI that evaluated it may make errors in similar ways, and results with fluent explanations may also be overrated. The information needed for reproduction and review is kept as the record of each study. Other researchers can review the results only if the task inputs and model version, the state of the tools and materials, and the settings and evaluation criteria are preserved. Reproduction matters not only for obtaining the same numbers again but also for tracing why results changed. If the variation across repeated runs, differences in how tasks are composed and the evaluators’ judgments are recorded together, the range of an effect can be explained. Publishing cases that showed excellent results together with cases that failed helps later research avoid repeating the same limits.

Responsibility for judgment

How is responsibility examined for judgments AI was involved in?

It is hard to set an appropriate scope of use from a list of technically possible actions alone. It should be possible to check who set the goals and constraints, what materials were provided and which person finalized the result.
The Scholars’ Council for AI analyzes these relationships but does not settle legal liability in individual cases by scholarly review alone.
It can be tested whether the evidence can be reviewed again when a user challenges a result, and whether the affected judgments can be found when an operator changes a rule. This is a way of checking the relationship between declaring a norm and making it work in actual procedures.

Cases in which norms conflict with one another can also be studied. There are times when the efficiency of work and users’ demands for explanation, or the sharing of information and restrictions on access, have to be weighed together. When different goals conflict, the reason for what was chosen and the other alternatives should be stated.

Is it enough to look only at performance?

For a purchase review system, looking only at the completion rate can make a method that quickly processes even requests with insufficient evidence look favorable, and looking only at the error rate can make a method that defers many requests look favorable. That is why whether completed tasks were appropriate, the reasons for and necessity of deferral, and the time spent on additional checks should be reviewed together. The cost of evaluation includes not only model usage fees but also preparing materials, human review, recovery after failures and interruptions to operation.

Changes to look at alongside technical results

Category	Examples of technical results	Examples of changes that remain with people
Accuracy	Lower error rate	Less harm caused by wrong judgments
Speed	Shorter processing time	Change in people’s review time and burden
Explanation	Rate at which evidence is given	Whether users can understand, challenge and correct
Learning	Performance with assistance	Change in the ability to perform independently without AI
Access	More users	Actual use and gaps among groups that need support

If it has not been confirmed what social change technical results led to, the two are reported separately.

Read more — Evaluating results and failures together

The main evaluation criterion should be set according to what is to be improved, and it should be examined whether improvement on that criterion worsens other results. Performance in real settings shows up across several criteria. Task accuracy and completion rate, processing time and cost, and the amount of rework and the review burden on users can all matter together. Setting out the evaluation plan before the experiment helps in interpreting the results. In the case of a purchase review system, looking only at the completion rate can make a method that quickly processes even requests with insufficient evidence look favorable. Looking only at the error rate can make a method that defers many requests look favorable. Therefore the appropriateness of completed tasks, the reasons for and necessity of deferral, and the time spent on additional checks should be reviewed together. The cost of evaluation includes not only model usage fees but also preparing materials, human review, recovery after failures and interruptions to operation. Failures are recorded according to their cause and the size of their impact. Errors in input materials, errors of interpretation, mismatches of authority, failures of execution and omissions in verifying results call for different responses. Publishing recurring types of failure along with average performance makes the usable scope clearer.

Evaluation reports should be connected to the four social goals of the Scholars’ Council for AI. Reducing harm is examined through the types and effects of errors, the right to judgment and accountability through whether review and correction are actually carried out, human capability through changes in learning and independent performance, and gaps in access and use through outcomes and burdens by group. For a goal that is hard to apply directly to a project, the reason is stated. If it has not been confirmed what social change a model’s accuracy or cost savings led to, technical results and social outcomes are reported separately.

Applying scholarship

How are scholarly concepts carried over into AI research?

Connecting scholarship to AI begins with reading the questions of the original research accurately. Deciding through which records and indicators an abstract concept is to be observed is called operationalization; there are several options in this process, and the reasons for a choice should be explained. When materials could not be found or a particular result could not be reproduced, it is important to state that status as well and not to turn what is unconfirmed into positive evidence.

Evidence from existing scholarship
Subjects · methods · results
Conditions under which it holds, and limits
Scope confirmed in the original research
Hypothesis for applying it to AI
States and actions to observe
What is expected to change
Claim to be newly verified
Comparative experiment
Compared with existing methods and no intervention
Effects · costs · failures
Empirical evidence from that experiment
Checked in the field and under other conditions
Differences in users · materials · settings
Scope of application and limits
Results and counterexamples
New questions and follow-up research
Explain counterexamples and differences in conditions
Connect to the next experiments and to theory

back to the hypothesis for applying it to AI

Existing research on people and organizations, new hypotheses for applying it to AI and evidence obtained from actual AI experiments are kept distinct. If the results do not fit, the mapping of concepts, the implementation and the conditions of application are reviewed again.

Read more — Turning scholarly concepts into working criteria

The work of connecting scholarship to AI begins with reading the questions of the original research accurately. It should be set out what phenomenon the research seeks to explain, which subjects were observed by what methods, and how far the results are supported. Deciding through which records and indicators an abstract concept is to be observed is called operationalization. There are several options in this process, and the reasons for a choice should be explained. Once the observation criteria are set, judgments and actions can be connected. When the absence of required evidence has been confirmed, it is decided whether to request additional materials, look for other sources or defer the decision. Each action has expected effects and costs. It should be indicated whether the explanation in the original research directly supports this action or was consulted in forming a new hypothesis.

Going back and forth between the interpretation of a concept and its actual working is the core method of applying scholarship. The record of applying scholarship keeps together the research question, the evidence in the source, the original conditions of application, what it corresponds to in AI, the new hypothesis, and the comparison conditions and results. In particular, the results of the original research and the parts newly interpreted by the researchers are recorded separately. When materials could not be found or a particular result could not be reproduced, that status is stated as well. It is important not to turn what is unconfirmed into positive evidence. Engineering implementation is carried out on the basis of this record. Scholarly explanations can be reflected in the classification criteria for data, in evaluation items and in the conditions for choosing an intervention. After implementation, the faculty review whether those criteria keep their intended meaning, and the Research Institute tests them.

Conditions and limits

Can research on people be applied to AI as it is?

Limits

Even if outwardly similar phenomena appear, whether they can be explained by the same cause is a separate question. The relationship of application can be written down in degrees of strength. Some research inspires the search for phenomena to observe, and some contributes to putting forward relationships between variables as new hypotheses; when an effect is confirmed through comparative experiments in an actual AI environment, empirical evidence for those conditions arises. If it is reproduced in several environments, the scope of application can be discussed more broadly, but these stages do not follow on from one another by themselves.

Read more — Conditions for applying research on people to AI

Observed similarities and hypotheses about causes should be kept distinct. When research on humans is applied to AI, the differences in the subject should be made clear. Emotions and experience, social relationships and institutions are involved in human behavior. Training materials and model architecture, instructions and tools, and the operating environment are involved in the outputs and actions of AI. Even if outwardly similar phenomena appear, whether they can be explained by the same cause is a separate question. The relationship of application can be described with differences in strength. Some research inspires the search for phenomena to observe. Some research contributes to presenting relationships between variables as new hypotheses. When comparative experiments are carried out in an actual AI environment and an effect is confirmed, empirical evidence for those conditions arises. If it is reproduced in several environments, the scope of application can be discussed more broadly. These stages do not follow on from one another automatically. The direct participation of scholars helps in examining these differences. Even so, the participating scholars’ interpretations should also be comparable with other evidence. Rather than the role of making a particular theory always right, the Scholars’ Council for AI takes on the role of bringing out the scope within which it can be applied and the phenomena that are hard to explain. In that process, AI research can gain new hypotheses, and existing scholarship can meet new questions.