The Role And Validity Of Proficiency Testing In A Laboratory’s Quality Management System

Catherine Brown is Vice President of
Collaborative Testing Services Inc. an ISO 1704 three accredited proficiency
test provider in Sterling Virginia she’s been employed by CTS for the last 18
years since 2015 she’s been responsible for overseeing both the nine disciplines
within the forensic Department and several other intern laboratory testing
programs she holds a master’s in Forensic Science and maintains
membership in the association of forensic quality assurance managers and
the American Academy of Forensic Sciences Laurel Farrell serves as the
senior accreditation program manager for ANSI national accreditation board work a
nap forensics accreditation program in Milwaukee. ANAB is a leader in the
accreditation of forensic science laboratories to the recognized
international standards miss Farrell began her work in laboratory
accreditation with the with a Strad lab in 2008 and continues to serve as a lead
Assessor for a nap plans assessment activities and she is an accomplished
forensic scientist in her own right Brendon Max was one of the one of the
subjects in this test program is the Chief of the Forensic Science Division
of the Cook County public defender’s office in Chicago the Forensic Science
Division is a group specially trained especially trained attorneys who
litigate forensic science issues on behalf of imaging clients mr. max has
extensive experience some litigation cases involving forensic
DNA and fingerprint evidence prior to his current position he was attorney in
the homicide task force in Cook County public defender’s office where he was a
lead attorney representing indigent clients charged with first-degree murder
Mr. Max is the author or co-author of an article on fingerprint proficiency tests
which will appear soon where he conducts training in the areas of crime scene
processing crime scene reconstruction blood stain pattern analysis alone with
case analysis in the area of blood stain pattern analysis and crime scene
reconstruction he’s a graduate of the FBI’s National Academy he’s a part-time
instructor in the graduate forensic biomedical program at Boston
University’s School of Medicine he is the past president of IAI the
International Association of Identification and served as their
representative on the consortium of forensic science organizations since
2008 and he is co-chair and treasurer of CFS oh good thank you yeah I’m real
excited about this panel I listened to a presentation by Brendan Max and the two
of his cohorts at the American Academy of Forensic Sciences and thought this is
a he raised issues that should be discussed in address so we haven’t come
here today to talk about those issues with respect to his ability to pass a
fingerprint proficiency test although he had never been trained in fingerprint
analysis so we’re gonna start off with him talking about fingerprint
proficiency test that he took in his observations about that then we’ll go to
Catherine Brown from CTS who’s going to talk about how they develop these tests
I mean how they determine the difficulty and how they’re processed and scored and
so forth the proficiency test that Brendan took was from CTS I’m very glad
that she is here to discuss that and then we’ll hear from Laurel Farrell to
talk about the role of proficiency tests in accreditation and why they believe
that it’s important under I so to have proficiency tests in laboratories and
then we’ll end up with this all-around good fellow next to me can who is a
practitioner and we’ll give the practitioners viewpoint of proficiency
tests he’s also as I understand developed proficiency tests as well and
one of his many jobs so I’d like to start off with Brendan if you could so good afternoon everybody I am really
excited to be a part of this panel I am really super interested in proficiency
testing maybe way more so than I should be but I became interested in
proficiency testing a few years ago when in Chicago and the public defender’s
office we started doing some kind of deep investigation of our local
fingerprint lab so in Chicago there’s really two crime labs that operate one
is the state crime lab which is the bigger more complete well-funded crime
lab system and then within our local police department there’s the remains
what used to be a crime lab which is now just a fingerprint division and what we
found when we were doing our kind of comparison of these two entities and
looking at the examiner’s fingerprint examiner’s and one versus the other we
found that while the examiners at the Illinois State Police received great
training and we seemed pretty competent to us and had good knowledge in the
field we could not say the same about the fingerprint examiner’s in our local
lab so are there any fingerprint examiner’s in here so what we found out
through our interviews is that they didn’t understand basic fundamental
concepts in their own field so for instance one concept that’s critical for
a fingerprint examiner to understand in order to distinguish a an actual match
from some some other fingerprint is the concept of a closed nine match a
fingerprint that looks like the source but is not actually the source there’s
no other important concepts like incidental similarity the fact that two
prints from different sources can just happen to look like each other randomly
we found out that our the fingerprint examiner’s in our local lab weren’t even
trained enough to understand some of the fundamental concepts in their own field
and yet what we found is that the fingerprint examiner’s in both labs year
in and year out repeat reported that they took
fingerprint for finished C tests and passed them year in and year out the
fingerprint examiner’s from the lab that seemed highly trained as well as the
fingerprint examiner’s from our local lab that seemed like they
were that their lab was substandard and that many other examiner’s were kind of
under trained so that’s what got us really interested in fingerprint force
proficiency tests interested enough that last year myself and two of my
colleagues paid for and took the CTS fingerprint proficiency test which I’ll
talk about in a few minutes but really proficiency tests have three really
critical important jobs in the criminal justice system right so results of
proficiency testing have been used to for Quality Assurance purposes right so
examiner’s take them labs can assess the results and decide when and where
retraining is needed that’s a critical part of of proficiency test you so swing
fast for instance says that yeah use proficiency tests for this purpose use
them to identify areas where improvements may may be needed so that’s
one really critical use of proficiency tests in the criminal justice system
another one is disciplined wide validity so you’ve got fingerprint examiner’s at
concluding these fingerprint examiner’s with the with the FBI published this
article in 2009 which states that decades of proficiency test results
fingerprint examiner’s taking proficiency tests doing really well on
them established that yes this method is valid and we can render
identifications or in this case they use the term individuals Asians because of
the history of our examiner’s doing really super well on these proficiency
tests and then maybe most directly from up for my cases our fingerprint
examiner’s testify in court I mean if you’ve ever heard any forensic testimony
whether it’s a fingerprint or otherwise you’ve heard an examiner say as part of
qualifications I have taken in past proficiency tests they put her right
there on their CV oftentimes at least with fingerprint examiner’s it’s the
only objective measure of their proficiency or competency at least for
examiner’s who aren’t certified right so if you’re not certified maybe these
proficiency tests are the only objective measure of your
of your proficiency in competency and then recently there’s been some
scholarship by Brandon Garrett and others who suggest that proficiency
tests are really the only if not the only the most critical data or
information that a court should look at when trying to decide whether to qualify
an examiner because these other benchmarks that have used before such as
experience and reputation are not really good indicators of competency so the
fingerprint error rate study is to show that there’s no real correlation between
for instance how many years you’ve been a fingerprint examiner and how you good
you do at these tests so brandon garrett and others suggest that courts should
turn almost exclusively to proficiency test results when looking at the
competency and qualifications i’m an examiner and trying to decide whether to
qualify an examiner as an expert so these are real three critical areas in
which proficiency tests play a big role in the criminal justice system but tests
that are too easy do not adequately serve any of these goals so if you’re
using proficiency test as your main quality assurance tool and that test is
calibrated to be so easy that everyone passes it their quality assurance
program will never will never identify breaking breaking points or points where
your examiner’s need to be retrained because it’s simply not calibrated
difficult enough to identify those locations those breaking points so I’ll
just let you know from my perspective including from having taken one of these
tests and I’m going to talk primarily about fingerprint proficiency test today
would have some reason to believe that kind of my belief on how easy these
tests are applies to some of the other tests in fact I’m in the process right
now of taking the DNA fingerprint for the DNA proficiency test and I’ll tell
you I’m finding it easier than the fingerprint one but let’s look at
historical data so when we first were interested in looking at how hard these
fingerprint proficiency tests were it was 2018 we look back to the last report
from 2017 and we asked ourselves what are the
results like where they show about passage rates here and in the 2017 cts
fingerprint test there is 97 percent of participants who got all 12 right 97
percent of participants who hast it that would be like 97 percent of high school
juniors getting a 1600 on the SAT that’s an ace all of them getting that right
when you’d look just a little bit closer 99 percent guide either a hundred
percent on the test or missed one of the 12 examinations when we looked back at
the historical data for looking back a few years for the tests we found these
kind of passage rates people who look who didn’t miss more than one question
on these fingerprint proficiency tests to all be pretty high so that’s to me
one good indication that these tests might not be rigorous enough so what we
decided to do was to take the test so who are we ah Who am I I’m an assistant
public defender in Chicago I have very little training in fingerprint
examinations in comparison I know some of the theory of it I’ve been trained on
the theory but I had never previously done a true fingerprint examination
which which means I had never been given a set of four exemplars from four
different people or three or two or any and had to do this searching function we
have a latent print and you’re searching across ten fingers and multiple palms to
trying to decide whether you can locate the source of that I had never done that
since I did work in the lab I had known of the software that fingerprint
examiner’s use so they can use software to quality Malka print to decide which
parts of print are high qualities where they should look for features and which
parts of prints are low quality so they should stay away those computer programs
also help them map out their features I had none of that
so I spent nine dollars on this jewelers little magnifier and I use that to help
me look at the printed out fingerprint images
and do my comparison and here are my results so out of the 12 questions
I got 11 correct I did miss one a false negative meaning I said that the for
this latent print it didn’t match any of the four people were provided when in
fact it did I think most kind of shocking to me is that the test wasn’t
complicated enough that I was forced into any kind of false positive
so never in the test was I given a comparison that was so close that I
picked the wrong one and it’s really sometimes the false positives that we
focus on in the criminal justice system I think maybe sometimes a little bit too
much but certainly certainly I did not stumble and into any false positives so
the question is did I just get lucky or was this test designed to be easy and so
going into the test so it’s not just the fact that that I passed this test that I
think is should be maybe a little bit alarming to the field but it’s the fact
that when you look at the data that was on this test and the questions and the
the comparisons I was asked to do it’s clear that this test was designed to be
easy so I knew that we would encounter some easy fingerprint comparisons that’s
the the green check clear prints with abundant features when you have a clear
print with abundant features the job is relatively easy to distinguish who who
the potential source of the print is but what I wondered is would we encounter
distorted prints one with few features those are ones that when it when
examiner’s encounter them in casework can sometimes be problematic
I wonder if with if I’d encountered prints that were made in blood because
some data shows that examiner’s have a hard time with those as well and that
they can be tricky I wondered if I were exam if I would encounter tonal reversal
prints which when examiner’s encounter those in casework they can be tricky as
well so an examiner total reversal means and examiner’s expectation about whether
the ridges are the dark or the light is actually the
opposite and they can get tricked into thinking that what is the Ridgid rich
and the rich pattern is actually actually the furrow which will take the
examination astray I wondered if I’d encounter any of those and then finally
I wonder wondered if I’d encounter any clothes not mattress so any situation
where I have to make a tough call between two potential sources of laser
printers that look very similar and it turns out that I didn’t really encounter
any of those in this test so first of all in terms of distorted prints and
prints with few features the data in the industry shows that although there is no
numerical threshold for features in fingerprint comparison there is a pretty
much a de facto threshold the research shows that around 8 or 9 features
examiner’s go from being uncomfortable calling an association to print to being
careful confident in that so that’s really about the threshold where if
you’re trying to give a challenging print you’re giving a print with maybe
around 7 8 or 9 features and you can tell from the number of features that I
marked for each of the prints where I made an identification that they were
all well above that there was no print that I encountered that was close to
that threshold no print that I encountered that had that was distorted
enough and had so few features that it became a difficult comparison for me we know that as I as I mentioned before
that prints in blood prints that exhibit tonal reversal are challenging ones in
fact in 1995 CTS issued a pretty challenging fingerprint proficiency test
this is one of their images and this is a print that was left in blood where I
could identify about five features maybe a skilled examiner might find one or two
more but I doubt that much more this is what a challenging fingerprint looks
like and by the way in this test the participants were also challenged by
what is known as a closed non match so the two prints at exactly that
participants head up here as the potential source we’re
from identical twins so the fingerprints were actually very close to each other
and these five features that I charted out here I could also find in both of
these prints in a similar constellation which would have made this comparison
pretty challenging for examiner’s and in fact was challenging for them because
the passage rate the error rate on that 1995 test was quite high in fact only
44% of participants on that 1995 test got all seven right so I didn’t
encounter any of those challenges in this test and we know that test makers
can design challenging fingerprint comparisons in the in the FBI black box
study which is considered maybe the preeminent fingerprint error rate study
they’ve had prints in there that were very challenging
they had this print in here that resulted in one false positive out of 30
comparisons in part due to the distortion there they also included this
print that had one false positive in 26 comparisons I think this one if I recall
right was the one that had some tonal reversal which made it again a more
challenging print so I did encounter these easy prints on the CTS test
there’s the easy ones but I did not encounter any of those types of
challenging prints that are encountered in casework in in every day fingerprint
comparisons in our jurisdiction so it’s not just that I got 11 out of 12 right
it’s not that I got lucky to me it’s that when you look through
the fingerprint comparisons that we were asked to do on this test it was an easy
test and it didn’t it didn’t include the types of difficult comparisons that are
encountered in casework it’s not just me that’s saying this or other research
this research paper came out last year at the same time when I was doing the
CTS test Henry Swofford formerly of the US Army
crime lab was doing a study of how rigorous or challenging CTS fingerprint
images are compared to case work they took 129 images from CTS
compared them to 225 215 prints from casework they used the FBI’s universal
latent workstation which will give a quality score of how good or bad the
prints are and what they ask themselves are are these prints that they
encountered in the proficiency test equally is difficult more difficult or
less difficult than the ones that you encounter in casework and what they
found was that the CTS tests were that were less difficult than the ones that
you encounter in casework so when you look at this chart this quality metrics
score of 30 along the x-axis is a relatively low score which means a
relatively challenging comparison on CTS tests over the years you got maybe about
10 percent or so of prints that scored that low whereas in casework you get
much higher than that at least in this case work sample of 215 cases so what
Henry Swofford and his co-author concluded was that CTS proficiency test
images are easier than those encountered in casework
there’s been other indicators for years that these tests have been relatively
easy so one of the things that participants can do are leave comments
when you finish your test there’s a comment section and you can leave
comments about what you think about the test and over the years examiner’s have
been telling the test maker that the test is easy so here latent prints were
not challenging enough the pattern is visible in every latent we would like to
have more challenging latent tests 2016 another comment along the same line
these tests could be more challenging maybe partial with lower quality and
clarity especially the palms could be more challenging they could they could
identify them right away in similar comments in 2015
that this test is at least for some examiner’s and the data shows probably
for most examiner’s not all of them not very challenging and then of course the
president of cts told the National Commission on forensic science just this
or about this in 2015 he said that easy tests are favored in the community in
the community meaning forensic science labs apparently like easy ones so that
is how they are generating so are these tests too easy I think there’s a lot of
information out there in the field that yes at least this fingerprint test is
too easy and probably others are as well and to me the accreditation process has
hasn’t done anything to kind of manage that or improve that situation so per
accreditation standards we know that labs are supposed to offer these and
examiner’s are supposed to take proficiency tests once a year these
tests according to accreditation are supposed to spot trends and identify
retraining needs a and a B is one of the major players in this space right they
are the big or one of the big not the biggest accreditation agencies for crime
labs out there and they identify approved providers and so CTS is one of
them but I don’t see anywhere where between CTS accreditation agencies other
professional organizations for instance and fingerprints like sweep fest are
doing anything to monitor how rigorous these tests so we have these really
critical uses of proficiency tests in the criminal justice system and we know
that proficiency tests may provide validity and may provide a QA system and
may provide a measure of competency for examiner’s in clear prints but it’s not
at all clear that is doing it’s doing the same for challenging prints that are
encountered in casework and this is especially this failure to challenge
test takers with closed non matches is especially critical and especially
disappointing because in our jurisdiction majority of the finger
prints cases that we get our APIs cases and if it’s cases are one where that are
ones where the distinction between the actual match and clothes not matches is
the most critical because it’s the job of a ‘thus to provide examiner’s with
three or four or five or ten prints that all look most similar to the latent
print out of all the prints in the database and because of that this
failure to test for clothes night matches in in proficiency testing is
very troubling because almost all of our cases are Ephesus so this 1995 tests
showed that examiner’s have problems with prints of blood on reversal clothes
not matches has retraining addressed this have examiner’s our examiner’s
competent today in these areas that maybe they weren’t before well I would
say that we don’t know because we stopped looking and because no one has
made examiner’s take tests that actually look for those types of challenges we
just don’t know the criminal justice system desperately needs qualified
examiners and reliable forensic evidence we have tons of evidence over time that
that hasn’t always been the case and we’re hoping to improve that I know
everyone here from defense attorneys or prosecutors to lab personnel want
reliable and and valuable forensic evidence in court one way to help make
sure that that happens is to improve the state of proficiency testing we have to
demand it courts have to demand it lab personnel have to ask for it thank you so first thank you mr. Melton for
contacting seats you have to be a part of this panel we believe that engaging
with all stakeholders is an important part in providing the best service that
we can to begin I so 1704 three discusses the role of proficiency
testing as follows the evaluation of laboratory performance through inter
laboratory comparisons using materials encountered in routine testing and what
is our role as a forensic proficiency test provider to provide homogeneous
case like samples and to provide the means to evaluate laboratory
performance homogeneity is extremely important in an inter laboratory
comparisons the variability of a participants results should lie
exclusively in their processes and procedures and should not be inherent in
the samples themselves let’s start with the first role of the proficiency test
provider providing homogeneous case like samples to do this CTS uses original
items as much as possible we are reluctant to move to the use of
reproductions when original materials are successful stable and appropriate to
the testing discipline reproductions sometimes referred to as artifacts are
used in almost all impression evidence as well as in handwriting blood stain
and gunshot residue these reproductions are used first and foremost due to the
inability to reproduce samples homogeneously such as with latent prints
handwriting samples shoes or tires our tests are designed to eliminate specific
variables we put a focus on case like evidence items and we’ll look to
standardize the known samples where possible our GSR test is a great example
of this as we provide reproductions of the known test buyers but provide an
actual evidence item for their own evaluation another interesting example
of the use of artifacts more than reproductions is our Firearms test we
cannot provide the firearm so instead we produce test shots from the known
firearm that are examined and lubed the actual firearm the sample choices we
make are not static they are always being evaluated for each discipline we
serve test sample performance and participant opinion are reviewed and
documented within our quality system we seek out expert guidance throughout our
test development and production process we solicit feedback from
discipline-specific organizations and accreditation bodies the advancement of
the technology within a dissapoint also drives our testing program I’ve been
with CTS for 18 years and the progression through the technological
advances within the forensic biology and DNA discipline has been fascinating the
evolution of DNA kits is clearly documented within
proficiency tests through the types of Lopes I collected and reported once we
determine the appropriate samples for testing we then need to look at the type
of evaluation style quantitative schemes are easier to evaluate however they are
rare in forensic sciences why are quantitative schemes easier because they
are more standardized iso thirteen five to eight defines the statistical models
that are used in inter laboratory testing worldwide the large majorities
of schemes utilize two confidence intervals 95 and 99 percent a 95%
confidence interval means that one out of twenty results will be highlighted on
average data no matter how good bad or ugly the data was this confidence
interval is usually seen as a warning limit if your results are indicated as
being outside of the 95% interval then a review should be performed a 99%
confidence interval will highlight 1 out of 100 results from that data set data
indicated as being outside of the 99% confidence interval should trigger an
action by the participant these schemes are common in high stakes industries
like medical automotive or aerospace in qualitative schemes CTS determines
consensus through a combination of expert evaluation participant responses
and expected results we have established a consensus baseline of 75% per test
item if that test item does not reach the 75% threshold it may not be used in
further performance evaluations as you can see I provided the consensus rate
per test item averaged over 5 years in some of our largest proficiency testing
schemes for both the full population and the accreditation accredited population
if the consensus rate in latent prints means the testing is too easy
than one about drugs analysis drugs analysis has not come under fire for the
science behind their determinations it is strongly instrument driven but it is
still considered a qualitative test is there a drug in a sample and what is it
so is the testing in this discipline too easy as well
once the evaluation style is determined then it can be utilized by the
appropriate parties for the evaluation of performance CTS is not a certifying
body therefore we do not grade proficiency tests the final
determination of performance is the responsibility of the laboratory and the
accreditation body we assist in that evaluation by providing the following
tools we highlight inconsistencies and provide consensus summaries in tabular
format let’s dive deeper into today’s example mr. Max’s team agreed with the
consensus for 11 of 12 items within the 2018 first half latent prints
examination test they all reported the same latent print as non identifiable
when the known individual was provided within the known samples the average
consensus rate per item for this test was 98 point for the inconsistent
results provided by mixture Mr Max’s team were highlighted the type of
inconsistency false negative versus false positive is not a factor in how we
mark inconsistencies from the consensus result all three reported the same
inconsistency and for this item the consensus rate was ninety five point
five percent CTS was at the af-s meeting in Baltimore this February where mr. max
presented his experience we had a staff and attendance at his talk and we had a
booth on the exhibit floor after his presentation we were sought out by
examiner’s who shared their impression of his discussion and they were shocked
that mr. max could say 11 of 12 was passing they knew it would be viewed
very differently in their lab triggering reviews possible retraining and would
have consequences during their testimony also in normal in a normal laboratory
setting the fact that all three test takers missed the same print would
definitely put their laboratories procedures into question as a
proficiency test provider our experience is that inconsistent results are a major
concern in laboratories no matter the number of inconsistencies or the type be
it false negatives or false positives they are investigated and can trigger
multiple layers of reviews with possible
retraining or procedural changes so our proficiency tests too easy
this question is continually asked and it is the wrong question instead we
should be asking do proficiency tests provide case like samples if proficiency
tests accurately represent the typical situations faced by a laboratory then we
have achieved the core goal of a proficiency test the consensus rate is
not an indication of difficulty and should not be looked at as such it is a
tool to compare a participants result to the general population for the
determination of that participants performance for that testing item as I
showed earlier in most of our tests we are seeing consensus rates with similar
values to the 95 and 99 percent confidence intervals used by most
industries outside of forensics however this is not by design we like the census
rates fall as they may the only time we take action is when it
is close to our consensus base rate of 75 percent then we investigate if there
was an issue with our samples the fact that untrained individuals can take
proficiency tests and return results that agree with the consensus does not
diminish the ability of proficiency tests if anything it helps show that the
underlying concepts found within that testing are understandable by a
layperson to include both the abilities of that discipline and the limitations
for example in the latent prints test that mr. Max and his colleagues took the
underlying concept is pattern matching pattern matching is a core concept in
many forensics fields be it by a human or a machine impressions tests firearms
tool marks GCMs results in drugs or even the electropherograms evaluated in DNA
all are based on the matching of similar patterns as long as we are providing
case like samples a lay person being able to understand these concepts of
forensic analysis is beneficial to all stakeholders the Center for statistics and
applications in forensics evidence see safe reached out to CTS to include a
survey with our 2017 second half latent prints examination test they requested
participants to rate each latent print in the test for perceived difficulty and
similarity to casework for the perceived difficulty question the scale went from
a 0 to 10 was 0 equal to the latent being extremely easy and a 10 rating for
the latent being extremely challenging for perceived difficulty the latent
prints in this test received an average rating of four point two seven for the
similarity to casework question the same 0 to 10 squid scale was used with zero
indicating of latent was nothing like casework and 10 for if the latent was
exactly like casework 4 similarity to casework the latest in this test
received an average rating of six point nine seven for those of you who like me
at times who are not statistically inclined a scale of 0 to 10 has 11
points which makes a score of approximately 7 fall into the top
quartile or top 25% of this scale as it relates to similarity to casework the
study also utilized the latent quality metric software available on the FBI’s
Universal latent workstation to rate the testing materials objectively CTS known
fingerprints scored ninety two point four and our latent prints scored a
seventy eight point six which both fall within the 90 to 65 range for
qualitatively assessed good prints so within this study were case like samples
provided yes the findings of this study matched cts s expectations the knowns we
provide were of high quality as I’ve already explained this is a deliberate
action by CTS to remove this aspect of variability in testing we recognized it
can be less common in casework but this is improving as fingerprint capture
technology advances reducing the quality of the knowns would achieve a lower
consensus but that is not our focus the latent prints provided were subjectively
rated at a seven and objectively at a seventy-eight another built-in aspect of
proficiency testing is that we only provide latent prints
which are of value for examination we recognize as an art it is artificial and
not like case work but we need participants to provide a response for
each item studies have shown that the value of identification determination is
also highly subjective which drives our need to provide higher quality items and
lastly these were digital productions not actual inked cards or lifted prints
for all of these key factors we are not surprised by a lower case like
perception and a higher lq metrics rating overall we feel that this study
shows for that test that moment in time we provided valid case like samples in
mr. Max’s presentation he showed us a chart from another recent study by
kortner and Swofford that used the l creek lq metrics to look at a very large
population of latent prints from CTS tests and compared them directly to a
set of casework latent prints at any point on that chart the proficiency test
items are rated no more than 10 points higher than the casework items and the
difference between the means as reported by the authors was thirty-six point six
three for casework versus 42 point two eight for CTS samples and this is on a
scale of 100 with the constraints on proficiency test that I’ve discussed
previously we were pleased with the results of this study as well so to
restate the scores that we have gotten both subjectively and objectively as our
items relate to casework are quite remarkable considering that the knowns
are purposefully of high quality the Layton’s are already determined to be a
value for identification and the evidence items are digital reproductions
not lifts now I knew I would not be able to ignore the concept of error rates and
proficiency testing but I only speak briefly about them and these are my
opinions our position is that proficiency tests are not the
appropriate vehicle for the determination of error rates in any
industry that we serve not just forensics however since forensics deals
more with qualitative subjective testing error rates determined through
proficiency testing would be extremely specific to the individual and the
sample agar these types of responses across
individuals samples and the spectrum of difficulty and case work that we are
discussing today makes it of dubious value
lastly the way that technology has been changing error rates determined today
will be nothing in 10 years maybe even less as we continue to automate
forensics testing image analysis software is going to completely change
this industry we won’t be looking at the error rate of individuals anymore but
the error rates of algorithms so where do we go from here
I’ve explained where we are now and the tools we use to continually evaluate our
services but we are always evolving the proficiency tests we produced in our
first 20 years look nothing like our proficiency test today just as the ones
we offer today will look nothing like our proficiency tests in the next 20
years CTS will continue to focus on utilizing
case like valuable and relevant samples within our proficiency tests we will
continue to be open to the community and all stakeholders we were the first and
are almost the only proficiency test provider to publicly publish our reports
and we’ve been doing so since 2000 we do our best to assist with studies see safe
was just the most recent and relevant example but when mr. max contacted us
for permission to use our images in his his paper we not only agree but we took
the time to provide him with high quality digital images so he would not
have to scan the photographs he received in the tests we attend important
stakeholder meetings and conferences I personally intended all but one of the
National Commission on forensic science meetings our forensic staff has a
presence not just as a vendor but as attendees for educational purposes at
almost all of the major forensics meetings and many that many of the
smaller regional conferences we respond to requests for interviews be it with
the PCAST or with journalists it is why I am here today and let me assure you
all of this engagement is not just for show we have in the past and will
continue to integrate appropriate changes into our proficiency test
designs will we seek to design a proficiency test with a specific
consensus rate No would we look to swap an easy latent
print for a more challenging one sure however this would not be in an effort
to raise our perceived difficulty rating of four point two seven but to raise the
perceived casework like rating of six point nine seven our focus will always
be on improving our ability to be more representative of casework
despite the acknowledged differences than to set an arbitrary level of
difficulty thank you very much for your time the
opportunity to gauge your community I got you fair look I got you okay good
afternoon everybody in ten I’ll add my thanks to you for including credit
aspect of proficiency testing as its when I think the remnants understand so
I appreciate that ok so front and center today is the
topic of proficiency testing and but I think it’s important to start this
presentation with acknowledging that at least from the accreditation perspective
it’s one tool in the toolbox it’s one piece in the puzzle and only one and
that many aspects of accreditation and the requirements that provide the
foundation for that support the overall quality system that’s in place at our
forensic service provider so um can ask me to address you what is the role of
proficiency testing in equality system and from my perspective and
accreditation perspective what proficiency testing adds is that outside
perspective it is a mechanism to compare forensic service providers the data
that’s provided does address the variability between forensic service
providers that perform the same service and importantly as Kathy says it does it
on a large scale it does it on a much larger scale than a
types of inter laboratory comparisons for some of the forensic disciplines
they have the ability to get a bit of this outside perspective through the
establishment of measurement traceability and the estimation of
measurement uncertainty but that’s not the case for all forensic science
disciplines so for some this is the only way that we truly can get that outside
perspective of performance so I I can’t talk about proficiency testing and
accreditation without talking about the history because there’s a long history
of proficiency testing and accreditation and that started with as cloud lab so
that’s the American Society of crime laboratory directors laboratory
Accreditation Board a different entity than as glad that you’ve heard mentioned
already today so a squad lab accredited the first laboratory in the United
States the Forensic service provider in the United States in 1982 and those
initial supplemental requirements included requirements related to
proficiency testing initially it was at just at the laboratory level initially
it started as a desired requirement in 93 the laboratory proficiency testing
requirement move moved to an essential requirement and in 1997 the second
requirement related to proficiency testing the one that dealt with
proficiency testing for each individual examiner was added as cloud lab did
create a proficiency test approval program they created that program back
in 1995 it was the mechanism then to evaluate the work that was being
performed by the proficiency test provider such as CTS accreditation of
proficiency test providers only began in 2010 a 14 sorry thanks Kathy so
accreditation of forensic service providers moved to
being based on an international standard fqs was the first accrediting body in
the United States to use iso/iec 17025 as the basis for accreditation in
forensics a squat lab moved their accreditation program to being based on
17025 in 2004 but in the 2005 version of 17025 there’s one mention of proficiency
testing it’s found in section 5.9 titled assuring the quality of testing
calibration results and you can read the class for yourself there in class five
nine one but at this point it’s referencing proficiency testing is
something that a laboratory may include as one of a number of options for
quality control procedures 17025 up the game in 2017 with the most recent
revision and for some of you that are not familiar with the standard they
totally remembered everything so now ensuring the validity of results is
found in section 7.7 you can read they’re in clause seven seven two that
there are requirements for the laboratory to monitor its performance by
comparison with results of other laboratories and it requires the
participation in proficiency testing and or participation in in inter laboratory
comparisons other than proficiency testing but when you read that
requirement there is no frequency that’s required there is also no requirement at
the examiner level it’s probably a good time just to talk about definitions a
little bit here and they are these are right out of 17025 proficiency testing
is a subset of the larger interlaboratory comparison all right
proficiency testing indicates that the performance will be against
pre-established criteria whereas interlaboratory comparison talks about
doing the comparison based on predetermined conditions okay so those
are two different things and then also it
discusses intra laboratory comparison so intra laboratory comparison is two or
more people within the same management system who would be using the same test
method testing the same or similar item inter laboratory now is going to put two
or more laboratories with different management systems different test maker
methods may even use different technologies to perform the same type of
tests to get the same test result they all serve valuable purposes but they’re
all slightly different all right one other aspect of this history of
proficiency testing and accreditation to look at is is the requirements of a
accrediting body and what this is going to tell you is again accreditation has
kind of been ahead of the game as a relates to proficiency testing so there
are requirements in iso/iec 1700 so standard that accrediting bodies are
held to if we want to be recognized at either the regional or the international
level and those have been there for a couple of decades now as well as in
policies by I lack that we must follow as a recognized accrediting body both of
these requirements predate the requirement in 1702 font so again at an
international level there’s a recognition of the value of proficiency
testing so how does all of this come together some of you may have seen this
before but we talked about this hierarchy of documents and how they
interrelate with the analogy of a multi-layered cake and if it’s my cake
it has to be chocolate so the bottom layer the cake is the iso standard so
1735 an AP forensic accreditation requirements form that next layer so
that’s where we add the forensic specific requirements the nab
accreditation requirements need to be read and conduct in conjunction with
17025 the next layer on the cake may the other standards that apply to that
Forensic service provider so they could be the FBI QAS standards or it could be
the American Board of forensic toxicology standards that you’ve heard
mentioned this morning the top layer that cake is the laboratories management
system where they will be writing policies procedures instructions methods
to actually implement all of these accreditation requirements and in that
layer we would expect them to consider state and local laws so let’s focus on
the a NAB forensic sector specific requirements here and take a look at
what we have in place today but really they haven’t changed that since the mid
90s so in a or 3125 clause 771 if you read that wording you will see
requirements at the location level or laboratory level as and at the
discipline level and you’ll see requirements both for proficiency
testing participation prior to being accreditation being granted and then on
an ongoing basis kind of the sister requirement to that is clause 774 and
this is the requirement that focuses an on the individual personnel of the
forensic service provider we have requirements for some type of monitoring
on an annual basis in each discipline in which they perform work the laboratory
requirement is for proficiency testing the personnel level requirement allows
for proficiency testing interlaboratory comparison in try laboratory comparison
or if more appropriate observation based performance monitoring clause 775 lays
some very basic ground rules for proficiency testing by the forensic
service provider things like ensuring that the results are not known or
readily available ensuring the use of the approved methods of the forensic
service provider establishing criteria for determining six
completion prior to the monitoring activity Kathy mentioned that the
criteria is really to be set by the forensic service provider it may be
appropriate at times to use the consensus based data and then relate
that to the laboratories normal expectations her performance in a
quantitative test the criteria set by the forensic service provider may be set
based on their estimation of measurement uncertainty and lastly here at clause 7
7 7 talks about the proficiency test provider themselves and the current
requirement is if available and appropriate that the proficiency test
will come from an accredited provider and if that is not available or not
appropriate because an EB does work on an international in an international
market and proficiency tests are not as readily available in some of the
countries where we are working today we certainly need this pathway for people
then there are other paths where they must provide an a/b what they’re going
to do information on what they’re going to do and that needs to be approved so
one other aspect of the history of proficiency testing here before we go on
I wanted to just bring in iso/iec 17025 intention to much today but we do have
accreditation of Forensic service providers in the United States based on
both international standards 17:02 o itself does not have any
specific mention of proficiency testing they do have a requirement for
monitoring of personnel the focus of this international standard is actually
on sight observed monitoring but later on this all of the traditional a NAB
accreditation requirements for proficiency testing at the discipline
level by location as well as the individual proficiency testing
requirements in all of the disciplines in
which the individual performs work so another important aspect of this is our
relationship with the proficiency test provider CTS and others a NAB receives
data on performance directly from the proficiency test provider we don’t have
the accredited forensic service provider send that to us we get it directly from
the p2p that data is reviewed by some of our staff as well as members of
proficiency review committees that we have established that data is reviewed
for trends it’s looked at and broken apart if different people are using
different instrumentation different technology can we see any advantages or
disadvantages strengths or weaknesses of different different types of
instrumentation or technology is there in fact a concern related or in
relationship to this proficiency test that a NAB wants to communicate to a
proficiency test provider so our staff and the proficiency review committees
serve that purpose for us so pros and cons right everything in life has pros
and cons and that’s true about proficiency testing to an TV forensics
does not look at PT testing as the be-all end-all by any means we do
acknowledge and are aware that some proficiency tests can more closely model
the testing process than others we do understand the limitations that a
proficiency test provider has to provide a large-scale product and control for
some of the variables that they need to control in order to limit if if there is
less than acceptable performance limit the cause of that but I will still stand
here and say that I believe all proficiency testing is useful and right
beneath that I will say I believe that all proficiency testing will can
you need to evolve Cathy talks about the evolution over her history in this
industry I certainly have seen it from all of the purchase insi tests I took as
a practitioner to my role now and it will do nothing but continue to evolve
and mature so what has a ne be done though in kind of with our knowledge of
the pros and cons of proficiency testing well we do have other NAEP forensics
specific requirements that I believe help support acknowledge again these
pros and cons we have specific requirements related to competency
testing that are also supposed to be practical examinations that cover the
spectrum of the work that that examiner will perform
generally speaking competency testings are going to be broader than proficiency
testing but even competency testing again is not everything I started here
and I’m going to end here proficiency testing is one piece of the puzzle that
is put together with all of the other accreditation requirements that become
part of accreditation based on iso/iec 17025 or 1702 oh and I think it’s
crucial that we remember that there are all of these pieces to the puzzle that
provide that framework to ensure the quality of the test or calibration
result reporting thank you very much so what I was to talk about is a
proficiency test taker again my background I was with Massachusetts
State Police for 33 years and was 13 years I commander the crime-scene
services section basically counts I would oversee all the impression
evidence so what we had was the latent print section footwear and tire track as well as cleansing we are accredited
under 17025 even when crime-scene unit so therefore
we would take proficiency is the director of the section would also take
cases – that’s the way out we would set up so therefore was taking proficiency
tests here a blatant print where have an analysis and also crime scene I’ve also
created a couple of internal tests in the area of latent print identification
as well as crime scene so having said that one of the things that one of the
slides that we would should have been there one of the first ones is what what
type of testing I think there’s a misrepresentation or misconception I
should say as far as what exactly a proficiency test is if you’re gonna
first of all if you are you want to give our proficiency test EPS comes in a
package and I can handle it off to someone maybe asthmatics received it late the impressions and I can try to
match them up to the Minoans on the other hand you can take Dennis
casework because one of the other things that you want to take a look at is is
your system your quality system so we insert that as an actual case so now
we’re actually not only looking with the examiner but we’re looking at the
examiner as he works through the system to ensure that he is complying with all
of our standards and then lastly one of the other things that we take a look at
that we stopped in Massachusetts we had to submit up prints into APIs and we had
to have some type of test to ensure that when I submit one the examiner submits
is blatantly to the æther system that they’re able not only to to be able to
enter the place but they’re also able to retrieve the results into print and look
at the results and if we have some closed non matches to be able to exclude
those closed on matches the other thing is as I mentioned before whether you
completed is face work and that’s the other thing if it’s a test are you
sitting down as you conceptualize a normal test or a regular cast and I’m
going to take that completion if you’re in fact taking that
as a test excuse me this case work one of the
things that you have to take a look at and there was out with human factors
publication some with years ago which I served on normally you’re gonna do your
comparisons the warning cuz your eyes do get tired so it’s not it’s not
inconceivable that over a couple days that would be looking at those images on
that package short that particular test so again how am I gonna administer and
how am I gonna take that test like why so you can take the blind can take you
this of case the images are put in there the labels to put in there yes the
labels from a potential scenario where you have your paperwork or you can a
cabinet applying test in other words your quality assurance versus quality
assurance section we take those images and insert them as a case likewise what
type of images are you giving it’s kind of interesting I have a study here I’ll
talk about comes the open Europe one of the things that they look about what
they try to test is when evidence comes in obviously thank you
images and then lastly the the comparison portion of it I do have a
slice of yourself again knowing that this goes all the way back to the
proficiency testing if you take a look at the National Academy of Sciences
recommendation back in 2009 when this was published they talked about
proficiency testing and the importance of proficiency testing so again as most
labs do as our lab did as well we have annual proficiency testing in those
particular areas where again you’re going to offer testimony if you take a
look at swig fast and as a member of hoops which stands for a couple of years
certainly they recognized the importance of proficiency testing and if you take a
look at the skull it’s intended to evaluate the individual and also the
agency’s procedures as I mentioned before and one of the
things that we were doing was testing their ability to be able to work their
way through the abus system or the aether system depending on what your how
you refer to it I will say this as far as creating a test the when we talk
about the close non matches those are very difficult to replicate the only way
you’re going to get a close non match is to be able to take some type of latent
impression when I say take some type of latent impression first of all you have
some type of privacy issues and then to come up with a closed non match unless
you have an identical twin and that prints many times are very similar but
you would have to put it into an 8-bit system to come up with a close non match
so it’s very difficult to create those likewise and again having created a
proficiency test tonal reversals are very difficult now you happen to see
that example of the tonal reversal on the board before where that was a bloody
impression and it’s not uncommon where if I was to see a total reversal they
would be in a bloody print so essentially some of the ridges have
blood on there kind of deposit likewise that wiped some of the blood off so when
I press down it’s going to be the blood and the furrows that shows up so mentor
it goes from ridges being read to furrows being read so to create those
it’s very difficult and where I’m going with this is as it is a test creator
internal proficiency test it becomes very difficult to replicate those
conditions and again as I mentioned before even entry if you take a look at
the European network of forensic science Institute’s they actually started
looking at this back in 2004 and they’ve tried to replicate some tests as well
with very similar results to what we have the test administration I talked a
little bit about this before it talks about annually these are the sweet fast
recommendations all conclusions must be those of the examiner without
consultation that’s how do you how do you administer this house going back to
you give them the examination once they had done with all the comparisons do you
have them turn it in do you have a verification as we talked about here on
five point four but then that must be documented again there are various ways
to do this I started to mention this before five point six is it open is it
in the open testing where the examiners are where they’re being tested even at
that is it submitted in a case format or is it administered just as a
test and here take this and work through it is it blind and that the examiner
doesn’t know it or is it double blind that the agency doesn’t know of the
testing so again there are different formats that you can construct or you
could use to test your system as well as your examiner even the National
Commission certainly the National Commission again we’ve certainly
recognized this or it has been recognized and pointed out again they
address the the need for proficiency testing and you can see down in the last
couple of bullets say they may be purchased they may be created externally
profusely tests may be created and reported internally as I mentioned
before the creation of an internal test again they vary between crime scene
and all of the other disciplines but if you think about it in the real world
when I’m trying to create a latent print test basically what we did is we had
some volunteers who were the positive prints essentially we’ve processed the
prints because we weren’t testing the individuals for their processing
techniques to develop the prints what we were looking for is to comparison
photography so what I’m getting at is to try to create those tonal reversals very
difficult but we’re able to create prints that are we actually rated them
as an ABC as far as the quality and what we did is and again what format and I
know we we had some comments on the latent lips depending on what your lab
receives in this particular case we had latent lips in the format that they
would receive the minimum when I talk about the format in this particular
click case black powder or metallic powder
likewise digital images because again to work a latent lift is different than a
digital image from a practitioner point of view so what type of format
are you using so in this particular case not only did I have digital images and
those were digital images that was submitted to a fist or I knew ground
truth as to whether or not I would get a candidate likewise on the latent trances
I have someone to deposit prints as we did we had plastic surfaces and again
substrate makes a big difference for clarity and when you take a look at
these prints and I know there was something some mention about the quality
we actually talk more about clarity now they’re synonymous but we’re getting to
clarity because it seems like people understand that a little better
so not only do we have the quantity of say level two information those rich
details of points as many of you may refer to them but we also have the
clarity of the image and again as I have someone randomly placed their hand down
I have no idea how the prints are going to turn out until we actually process
those surfaces so we try to do is the best we can and I know there was some
mention about having prints of no value well the problem there is when you think
about it as soon as I have this blot and let’s say there’s no value essentially
the examination is concluded at that point for the print I would much rather
have a print that’s of value because now they can search the five or six
different individuals and to test their ability or their aptitude to actually
compare it and find that matches in fact it’s there I’ll do the exclusion so to
have non matches I know this in some literature and even antfee talks about
having some prints of no value in there I would think you’d want to have some
prints where you actually have an individual or person that you could
associate with associated with a PCAST and again just going through some of the
historical aspect here they’ve certainly talked about proficiency testing and
even the American the triple-a s study talked about the proficiency testing and
the need for proficiency testing as well and I saw when we’ve heard enough about
ISO standards but again as I mentioned before the external internal what are we
testing the examiner you’ve heard me some mention that before how will we
administrate administering the test the test duration again is it going to be
over a couple days is it going to be again a sit-down session but what I did
want to talk about as well is also the comment one of one of the things that
you’re going to find is an examiner and specifically those comments were
pointing out as to whether or not the test is easy I can tell you right now is
an examiner having trained examiner’s assigned to this section there are
certain aptitudes that people have okay and I can tell you that I can have
a portion of a lady and they have been there was a written article written up
2017 where they’re actually no level two characteristics and points it was ID’d
on level three detail it was out in Illinois as I believe her Indiana but
what I’m getting at is there are certain individuals who like some pride in the
work they do I can give them a portion of elation or a fragment of elation and
they can act automatically orient that and say you know what I’m looking at the
right side of the finger I’m looking at the left side of palm and many times
when they do take these counsel again having been with qualities quality of
the director of that particular lab at the time if I get the exams back when
they would take the exams certainly some individuals find it very easy you know
when they pride themselves and I I found it it was pretty quick and there others
that may not necessarily struggle but they take it time and again depending on
the at least their assessment of the quality of the late and they may work
that through so some of those statements in there don’t necessarily reflect that
and as I just mention this that entity study that I have back they they started
in 2004 and they did these six different testings of collaborative exercises and
different countries actually put it together in 2004 they actually had the
first exam it was put together by Estonia and essentially they had several
false positives in there and again erroneous identifications one of the
things that they did find and that’s another thing you need to consider when
you’re putting together a proficiency test is how much information does that
person get you know there was always out there that say they shouldn’t have any
information they just get this and that’s all we have but in this
particular case when you have a non-porous surface this print happen to
be on a piece of plastic and happen to hit touch or come in contact with
another surface and it is possible at a print can transfer is it it does happen
it’s called lateral reverse prints so essentially what’s right is left and
left is right in this particular case one of the latencies have that
phenomenon occur so it doesn’t happen and as they passed out the test it
wasn’t until after the tests had been taken where they had these erroneous
conclusions and again in the event or in this case if they would have told the
examiner that this happened to be on plastic the examiner may have considered
the fact that you have this possibility of lateral reverse prints so again it’s
not as easy as one may think that you put these exams together and hand them
out because again to be able to replicate some of the conditions that we
see in casework and I would say that tonal reversals and again I’ve been
around for a little while you don’t see those very often close non matches
absolutely in fact I advocate we have this case come up in Massachusetts a few
years ago with the close of 2003 okay well it’s not like what tell you
the place I keep watching with Parker the closing on matches I would just say
this the computer creates an algorithm it’s not as good as a human what we
found is in those closing on matches when you actually pulled them up I’ve
done that on occasion because we have a court motion and when we started the
poem up was very clear once you started to look at the latent ink and my
suggestion to the prosecutor was you know what let’s give them all the
clothes non matches and we can show the jury how far off the computer is is that
how close the computer was and in this particular case it depends upon your
algorithm numb to my last 30 seconds so thank you very much

Leave a Reply

Your email address will not be published. Required fields are marked *