Does AI use the right law for the right jurisdiction?
Caribbean Legal AI Reliability and Jurisdictional Reasoning Study · Proposed research program
Support the research · Explore the study
The question
We propose to examine whether AI-generated legal answers select the correct jurisdiction, rely on supporting authorities, respect the relevant legal date, and identify missing facts. We would also examine whether access to appropriate legal sources improves reliability.
Three jurisdictions, distinct legal systems
Proposed scope
Jamaica, Barbados, and Trinidad and Tobago, with an initial focus on employment law and civil and commercial procedure. The study would use synthetic scenarios rather than client files.
How the research would work
Synthetic legal question
A specified jurisdiction, legal date, and set of facts.
AI answers
Responses under different access conditions for legal sources.
Blinded expert review
Correct jurisdiction · supporting citations · relevant date · missing facts.
Public evidence
Reviewed aggregate findings, methods, uncertainty, and limitations.
Local legal expertise would verify sources and assessment materials. AI answers would be compared under different source-access conditions and graded without reviewers knowing which model produced them. Methods, uncertainty, and limitations would accompany reported results.
Research roadmap
Develop
Refine the protocol, recruit local expertise, and secure source permissions.
Review
Prepare synthetic scenarios and obtain legal and methods review.
Test
Compare AI answers and assess them through blinded expert review.
Share
Analyze results and publish reviewed findings with their limitations.
Timing depends on funding, reviewer availability, and permissions. These stages describe the proposed work, not completed results.
Current stage
The protocol and evaluation infrastructure are in draft development. Expert review, legal-text permissions, institutional arrangements, and model testing remain to be completed. Simulated software checks are not research findings.
What we intend to share
We plan to share a methods report, aggregate findings, limitations, and accessible explanations following review and release approval. Protected test materials and restricted source texts will remain controlled.
What the study can tell us
The research is intended to produce evidence about performance under documented conditions. It will not provide legal advice or certify a model as safe for legal practice.