Of Oversight

Against the Fifth Danger: the Unbound.

1. No system powerful enough to do grave harm shall be released until its makers have shown, to reviewers independent of them, that it can be understood well enough to be trusted, and stopped if it fails.
2. The more power a system holds over people’s lives, the higher that bar. And the review does not end at release: it continues as the system changes and as new risks come to light.
3. Every powerful system shall be open to independent testing. Its makers shall publish what was tested and what was found, holding back only what would put people in danger, and showing what is held back to the independent reviewers.
4. There shall always be a way to stop it, held by people who answer for it, and a plan to stop it safely, so that those who depend on it are not harmed by its stopping.
5. Those who warn of danger from within shall be protected, not punished.

By these signs it is kept

• Powerful systems are tested by reviewers independent of their makers in pay and in power.
• The results are public, and what is held back is explained.
• The way to stop them has been tried in practice, and someone answers for it.