validate_the_work_of_coders [PCAT Online Help]

- PCAT project managers can review and validate the work of coders

PCAT project managers can review and validate the work of coders

During training sessions, you may want to review and validate the work of coders to make sure they understand the requirements of the classification task. The original CAT Toolkit was built primarily for this purpose. Adjudication by consensus or expert validation is a key feature of this platform.

To begin validation, the dataset must be locked. To lock a dataset, select View Datasets from the Datasets drop-down menu at the top of the screen. Click on the name of the dataset, then select Lock Dataset at the bottom of the right sidebar. Once the dataset is locked, select “Validate Dataset” from the “Validations” drop-down menu. Select a dataset from the drop-down list to begin. At this time you can also select which specific codes you wish to validate if you do not wish to validate all codes.

You will be shown a document as well as the code(s) selected for it. To validate, either click the Valid button on the bottom of the screen or type 1 or Y. If the coding is invalid, click Invalid or type 3 or N. Using keystrokes is efficient when validating large datasets, but you can use whichever method you prefer without changing the result. After validating the document, the next document will load automatically.

Editing Validations

If you make a mistake when validating, you can still fix it. Select View/Edit Validations from the Validations dropdown menu at the top of the screen. Click on the name of the dataset you want to edit, then you can select which coders or codes you want to review. Click on the icon to change your validation.

Coder/Dataset Statistics

Coder Statistics - This tool shows the coder's individual statistics. By choosing a coder, PCAT will provide information about them in the following order:

# Datasets - Total number of datasets the coder has worked on
# Annotations - Total number of codes applied by the coder
# Validations - Total number of validations made by the coder
% Correctness - The total percentage of occurrences where the coder's annotations was a valid answer

Dataset Statistics - This tool shows statistics for a specific dataset. By choosing a dataset, PCAT will provide information in the following order:

Total Valid Answer - The total number of valid answers made by all coders
Validations by coder - Total number of valid answers made by each coder
Validation by code - Total number of valid answers for each code

Most Frequently Asked Questions

Why would I use this system? | Where do I get FDMS bulk downloads? | Does PCAT identify duplicates? | What is QDAP?

© 2009 - 2010 Qualitative Data Analysis Program (QDAP), in the University Center for Social and Urban Research, at the University of Pittsburgh, and QDAP-UMass, in the College of Social and Behavioral Sciences, at the University of Massachusetts Amherst. As of 2010, PCAT and this PCAT Help Wiki are maintained and improved by personnel from Texifter, LLC, which is a software start-up located in North Amherst & Springfield, MA and online at http://texifter.com.

Content on this website was made possible with the following grants from the National Science Foundation: III-0705566 “Collaborative Research III-COR: From a Pile of Documents to a Collection of Information: A Framework for Multi-Dimensional Text Analysis” and IIS-0429293 “Collaborative Research: Language Processing Technology for Electronic Rulemaking.” We are also grateful for financial support from the U.S. Environmental Protection Agency and the U.S. Fish & Wildlife Service. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the National Science Foundation.