THE EVALUATION OF DOCTORAL THESIS. A MODEL PROPOSAL

de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
e-Journal of Educational
Research, Assessment and
Evaluation
Revista ELectrónica de
Investigación y EValuación
Educativa
THE EVALUATION OF DOCTORAL THESIS. A MODEL
PROPOSAL
[La evaluación de tesis doctorales. Propuesta de un modelo]
by/por
Article record
Ficha del artículo
de-Miguel, Mario (mario@uniovi.es)
About authors
Sobre los autores
HTML format
Formato HTML
Abstract
Resumen
The aim of this work is the development of a model that
can be used as a framework for evaluators and PhD tribunal members, either during the previous administrative
processing phase or at the actual exposition and viva on
doctoral dissertations. To this end, once reviewed the existing literature and normative, a new proposal is laid out
on the main aspects and criteria to be analysed and assessed from any given scientific work, thus introducing a
tool that facilitates and specifies this work. Our model is
intended to be an open proposal on which we would welcome any contribution and suggestion.
El objetivo del presente trabajo es la elaboración de un
modelo que pueda ser utilizado como marco de referencia
por los evaluadores y miembros de tribunales de las tesis
doctorales tanto en la fase previa a la tramitación administrativa como en el acto de exposición y defensa de la
misma. Para ello, una vez revisada la literatura y la normativa vigente al respecto, efectuamos una propuesta sobre los principales aspectos y criterios que deben ser objeto de análisis y valoración en todo trabajo científico, y
presentamos una herramienta que facilita y precisa esta
tarea. Nuestro modelo constituye una propuesta abierta
sobre la que esperamos aportaciones y sugerencias.
Keywords
Descriptores
Doctoral thesis, Doctoral dissertations evaluation, Doctoral thesis quality, doctoral dissertations evaluation
model.
Tesis doctoral, evaluación de tesis doctorales, calidad de
las tesis doctorales, modelo de evaluación de las tesis doctorales.
Introduction
One of the tasks we university professors
frequently have to undertake is the assessment of written work presented- beyond
formal procedures –in order to ensure their
quality as doctoral theses. It is not customary
for the universities to provide detailed protocols so that the assignment can be performed
following a series of criteria that are considered a priori fundamentals for the elaboration of the mandatory reports. The rule is to
employ loosely structured forms where
evaluators make judgements agreeing with
their particular logic about how it should be
and how he/she should present a scientific
work so as to qualify as a doctoral thesis.
This lack of a clear and common framework
about the criteria to be incorporated when
assessing a piece of work of these characteristics implies that the reports issued are imprecise and impractical when taking decisions.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 1
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
The lack of precision resulting from this state
of affairs when issuing reports about a thesis
is well-known: the majority of mandatory
reports for the administrative processing of
theses is positive – the number of rejected
theses does not even amount to one percent –
and the vast majority of them are awarded the
highest grade – approximately 92% of defended theses –although the quality of many
is questionable (Valcárcel et al., 2002). This
uncommon distribution of results in the assessments of doctoral theses shows that either
the protocols implemented are not suitable, or
the assessment is not carried out with the
scientific rigour desired.
From a technical perspective the problem
stems from the lack of clear and explicit criteria that should be taken into account when
making required judgments. The latest Spanish legislation about – Royal Decree
1393/2007, September 29th – determines that
“the university establishes procedures with
the aim of ensuring the quality of theses in
terms of formulation and the evaluation
process” (art. 21, 2). It states in the same
text that “the doctoral thesis will be evaluated in a viva to be held publicly…and the
examiners will issue a report and award an
overall grade to the thesis according to the
following scale...” (art. 21, 6, 7). The meaning of the rule is clear: the procedures and
criteria to evaluate the theses constitute an
internal problem of the universities that
should be regulated explicitly by them [1].
To date, the regulations established for doctoral programs and theses for various universities seem to address administrative questions rather than academic ones. It is unusual
for universities to establish detailed regulations concerning criteria and procedures to
assess the quality of the processes involved in
developing a doctoral program, such as the
products it generates, i.e. doctoral theses. It is
customary, though, to focus the whole
evaluation of the doctoral programs on statistics and quantitative indicators about entry
and exit dates without analyzing the proc-
esses and causes involved in the dropping out
or the considerable delay in the progress of
the students. They neither require specific
criteria to be considered regarding the quality
of a thesis, such as its contribution to scientific knowledge on a particular field of research, or whether it might solve practical
problems of social reality.
As previously stated (Valcárcel et al., 2002),
the final stage of assessment – focused on the
quality of the thesis – comprises three steps or
moments that should be scrutinized, i.e.: the
previous assessment that each university establishes so as to authorize the administrative
processing; the subsequent assessment made
at the time of public presentation and defence
of the thesis; and the assessment that can be
completed thereafter of the publications and
applications derived, as well as awards,
grants, projects and other achievements generated. Obviously, the work to be done in
each of these phases has different aims, and
therefore requires different mandatory forms
for each type of evaluation. It is therefore
necessary to clarify the different approaches
to these assessments based on the data and the
type of report that the evaluator is asked to
produce in each case.
This is the issue we intend to address in this
article: to present a model that can establish a
framework of tasks to be performed by the
evaluators who are required to assess doctoral
dissertations. Thus, we have made a revision
of the relevant literature on the nature and
purpose of doctoral theses in order to synthesize and structure the main parameters and
criteria that define the quality of this type of
work, as well as their contribution to scientific knowledge. Additionally, we have elaborated a comprehensive model of the various
criteria to be included in the evaluation of the
thesis, which provide objective judgments
and can be deployed either fully or in part
depending on the type of evaluation required
in each case.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 2
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
2.- Prior assessment of the doctoral thesis
reports.According to the regulations cited, the legal
text only specifies that “in the evaluation
process and prior to the defence/viva, the
university should guarantee advertising the
final doctoral thesis so that other doctors can
submit comments on the contents” (art. 21.4).
To date, the procedures implemented in distinct universities to carry out this assessment
are very diverse. They are rarely carried out
by a rigorous and independent evaluation to
determine objectively the quality of the work
and its contribution to scientific knowledge.
Likewise, it is not a good protocol to make
this assessment prior to comprehensively and
objectively making pertinent decisions on
your processing. If the assessment procedure
is not thorough, the result is predictable: the
majority of these theses evaluation ends up
being more of a ritual than an authentic
evaluation.
It should be remarked that the administrative
processing of a thesis involves the supervisor’s approval to proceed with its public defence, the customary report or reports required by the Department hosting the thesis, a
period of public exposure so that interested
doctors can submit pertinent arguments, and
finally, acceptance to proceed with the defence by the University Doctoral Committee.
All these processes can be considered evaluating actions or strategies, although in practice their effectiveness is reduced to a minimum. Once the thesis is given the go ahead
by the supervisor, the Doctoral Committee
follows suit and the panel of examiners appointed on the supervisor’s suggestion is
most likely to be ratified. This means that a
priori the quality guarantees of any given
thesis rest on the prestige that the university
has, the involved Department, the Doctoral
Program studied, and especially, the recognition that as a scientist the self-same supervisor holds (López Yepes et al., 2008).
Although this is normally the standard procedure, there exist different ways to proceed in
other scientific fields whose interesting alternatives are worth considering because they
can contribute to helping solve the problem at
hand. We will focus on some of them:
1. Assessment linked to the research group or
team. – Some university departments demand
that the entire process of thesis writing –
from the development of the project to the
writing of the dissertation – be supervised by
a collegiate research group or team made up
of at least three competent researchers in the
field of knowledge involved. The previous
evaluation is therefore conditional on a favourable report by the investigators appointed
– together with the supervisor – to ensure the
quality of the writing process as well as the
quality of the results obtained, since they are
acquainted first-hand with the project and the
work carried out by the doctoral candidates.
The advantages of this strategy are unquestionable for the doctoral candidates, since it
allows for the assessment and contribution of
the supervisor and the rest of the members
that make up the supervising group, while
also constituting a guarantee for the institution.
2. Assessment linked to a pre-reading session
of the thesis. – In the case of well-established
research groups and institutes, once the thesis
is completed, the doctoral candidates perform
one same presentation for all the research
specialists in the group or institute so that the
student can receive feedback in the form of
final thoughts, opinions, and suggestions on
their work. Not only does this preview or prereading of the thesis encourage the exchange
of information, but it also increases the possibilities of spreading the work in national
and international scientific networks. A variation of this evaluation procedure consists of
complementing the pre-reading session with
the mandatory reports of two external independent reviewers / examiners.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 3
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
3. Assessments by means of external experts’
reports. – Recently, various universities have
established as a prerequisite for a thesis acceptance to proceed that two external evaluators issue their favourable reports. Although
this assessment method should be given due
attention, the elements involved in its development do not guarantee its efficiency. It is
necessary to rigorously reconsider some of its
weak points: appointment system of the
evaluators, their competence and anonymity,
protocol criteria for completion of assessment, degree of interrrelatedned of the reports, etc. - since in many occasions it ends
up being more of a formality than an authentic evaluation. In short, establishing an external evaluation system cannot be enough
unless measures are taken to address the possible drawbacks posed above (Valcárcel et al,
2002).
4. Assessment linked to specific criteria of
quality. – The fact that nowadays there is
widespread access to scientific works by
number and type, some university departments have established a preliminary requirement: to grant a favourable report in this
initial evaluation when the doctoral candidate
has published two or three works on the thesis topic in journals, or has presented papers
and/or attended congresses or conferences
related to the scientific field in question.
Commonly, under this circumstance, neither
the pre-reading of the thesis nor the revision
by external experts takes place. It should be
remarked, however, that such a choice, constituting itself a new concept of what a submitted thesis should be, is not very common.
In fact, in some academic fields the research
piece is expected to be unpublished, its diffusion being carried out once the degree is
awarded (Sánchez Rodríguez, 2002; López
Yepes et al, 2008).
The shared purpose of the options discussed
above is to ensure the minimum quality of the
research that is to be defended as a doctoral
thesis. Given the usual lack of response from
doctors during the public exposure period
established by regulation and the purely bureaucratic role of the Doctoral Committee to
this end, universities should establish some
kind of "filter" that may offer the guarantee
needed in order for the thesis to be given the
go ahead. Perhaps the option of having a preliminary evaluation by two experts from outside the university, not included in the panel
that has to judge the defence, would be the
most appropriate choice as long as it is done
with sufficient transparency; and their evaluation is made by applying a standard protocol
in which comments are made explicit in an
objective and binding manner.
3.- Evaluation of the Viva and Public
Defense.The present Spanish legislation does not include any novelty about the composition of
panels that are to judge the thesis in the public defense. The legal text states that “the
examiners that evaluate the thesis must reach
an agreement employing the rules established
by the university. All members must have a
doctoral degree and confirmed accredited
experience. In any case only two members
from the University responsible for issuing
the degree can take part in the viva”. Furthermore, when a doctoral thesis has been
selected for a “European mention”, the board
of examiners should include an external doctor pertaining to some higher educational
institution or research center from a European
Union country other than Spain.
As previously indicated, the universities must
continue to have the final word in the selection and appointment of the members that
will conform the examiners’ board. Thus, it is
the universities’s duty to establish a procedure for their appointment that offers the desired credibility and grants their independence when evaluating the thesis. To this day,
since the university proposals deployed to
appoint viva panels stem from those of the
thesis supervisor, these panels cannot be regarded as completely independent due to the
networks that have been established between
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 4
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
the investigators at various academic levels.
The standard procedure of enclosing a brief
CV of the intended examiners has not provided enough evidence in establishing the
suitability and competence of the many
members of the proposed panels.
No specific regulation exists about the criteria of evaluation that must be followed by
examiners in order to grant a mark. The only
general specification is that “the panel will
issue a report and grant an overall grade to
the thesis in accordance with the following
scale: fail, pass, “good”, and excellent. The
panel will award the distinction “cum laudem
(with honours)” if the overall grade is excellent and has been reached unanimously. The
standard practice is for the examiners to make
an individual assessment prior to the viva
according to their beliefs about the parameters that such a piece of scientific work must
meet. Equally, a first assessment of the thesis
is made, focusing on the points that are less
clear or are open to debate – according to
ritual –, but without questioning or analysing
the work following any systematic criteria
previously established.
Regarding the thesis defence, a number of
criteria exist which are also open to evaluation. On the one hand, various authors insist
on the doctoral candidate’s ability to communicate the contents of his/her work clearly
and rigorously, using the appropriate language of the field of study, as well as any
graphic materials so as to get the message
across to the examiners and the rest of the
audience present (García de la Fuente, 1994;
Sierra Bravo, 1999). On the other, provided
that we are in front of a scientific event, it is
paramount that the candidate should be able
to argue appropriately the findings obtained,
any advance in the area of knowledge, and
the solutions to the problems formulated. In
all, the doctoral candidate must show his proficiency and scientific rigor as a researcher,
which may be elicited from the answers to
the questions or objections posed by the examiners; his own critical assessment of the
thesis; and the effective proposals advanced
about new lines of research in the field (Carreras Panchón, 1994; Desantes & López
Yepes, 1996).
It should be noted that the defense of a doctoral thesis constitutes a fundamental part of
the academic process, whose purpose is to
confirm the competence of the doctoral student. In this event, the board represents the
scientific community and, as such, must
judge the completed work of the doctoral
candidate, make the appropriate objections or
suggestions. The defense will contribute the
necessary evidence so as to sanction the research contributions as scientific while allowing the candidate to show his/her suitability
as a researcher. For this reason the defense
cannot be regarded just as a formality, but as
the public evaluation of the thesis from the
scientific community, as well as part of the
scientific training of the doctoral candidate.
However, such a view is not upheld widely,
for the defense of the thesis normally amount
to a social rather than a scientific event.
Neither does a precise rule about the procedure for awarding a grade to a thesis exist.
Apart from the comments and criticism –
more or less appropriate – completed by the
members of the panels, the grades are not
discriminating. The data gathered from another work (De Miguel, 2009) reflects an
atypical distribution of grades. Since the majority of the theses get the highest distinction,
this should be the target of a detailed revision. From our point of view the problem lies
in the fact that the universities do not require
examiners to use detailed protocols in which
all of the relevant questions to evaluate a thesis are being collected. This results, in practice, in most reports being prepared in a very
generic manner, avoiding the most critical
aspects. Sometimes, to avoid contradictions,
the report is issued once the grade has been
agreed upon. Hence, it would be desirable
that any decision about the criteria be formulated according to a system of scales or quantitative and qualitative ranges, provided their
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 5
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
effectiveness when deciding about the grading.
In conclusion, we believe that the process of
evaluating a thesis should be rethought thoroughly as regards the procedure for appointing the viva examiners so that the highest
degree of suitability and independence is
reached; and as the introduction of criteria of
rationality in the evaluation protocols to be
deployed when making the mandatory reports
on the thesis at the stages of processing and
defense. The use of protocols where the criteria are presented objectively through quantitative and / or qualitative scales allows for a
better match between the prior assessments
and the grades, correcting, thus, the atypical
distribution currently showing, while facilitating the production of argued reports whenever deficiencies are detected.
4.- Assessment of publications derived
from the thesis.We have already indicated that it is necessary
to carry out an assessment of the thesis subsequent to its publication with the intention of
measuring its true contribution to scientific
knowledge and to the resolution of practical
problems. For this purpose, the contributions
and applications derived from the thesis
through publications, patents, protocols, presentations and projects must be monitored over
time. Even though this evaluation of the impact of the thesis is not an easy task, it should
be a priority, since it is the fundamental criterion considered when special awards, projects, scholarships, and other forms of recognition are granted to both the authors and supervisors of the thesis. This assessment of the
thesis’s contribution is also important in
strengthening the quality assurance systems of
doctoral studies (De Miguel, 2009).
Some authors, however, believe that this impact assessment should be taken into account
in the thesis defense to the extent that essential parts of the thesis have been published or
accepted for publication in prestigious jour-
nals. They also believe it necessary to assess
the dissemination of findings made through
conference presentations, seminars, or invitations from national and foreign research centers (Sánchez Rodríguez, 2002). To this end
the doctoral candidate is requested to attach
his resume to his defense so that the panel
can evaluate the impact that the thesis has
had up to that point, and that which can be
expected of in the future. An initial assessment of the impact of the thesis should therefore be made at the moment of the thesis defense, though logically this type of assessment should be extended to all theses systematically and over a determined period of time
(eg. 5 years).
Among the possible means of disseminating
the ideas of a thesis, scientific publications,
“those that constitute a real contribution to
the progress of scientific knowledge and
technical development and that have been
disseminated by an adequate type of medium
within the scientific community,” provide the
most immediate channel. It should be noted
that in order for a publication to be considered scientific, it must contribute to the advancement of knowledge within a discipline
or about a social problem and be disseminated through media that is subject to
anonymous and independent internal control
by experts. The Scientific Committee of the
publishing medium chosen —journal, publishing company, etc.—guarantees in this
way the quality of a work for its publication,
as not all publications have the same scientific relevance.
In the fields of the Social Sciences and Education, preferential value is given to contributions published in journals with a longstanding reputation, accepting as such those that
occupy leading positions in databases like the
Journal Citation Report of the Social Sciences Citation Index (Institute of Scientific
Information, Philadelphia, PA, USA). The
gauge employed by this institution to evaluate the quality of a publication is an “impact
factor” constructed by averaging the citations
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 6
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
received by a publication in a particular journal. The impact factor is calculated by counting the number of citations a publication receives in a particular journal each year, using
the two previous years to determine the impact factor of a given year. The greater this
impact factor, the greater the quality of the
publication (De Miguel, 1997).
The use of this gauge, while widely accepted
and frequently used in other scientific areas,
causes difficulties when applied to the field
of the Social Sciences and Education, given
the peculiarities of research in this field, the
diversity of existing publications, and the
lack of specialized computerized databases.
Though we have recently made progress in
this area and now have some databases that
collect publications regarding education, we
must recognize that we are still far behind
other scientific fields in which there is general knowledge of which publications and
databases are most prestigious and what requirements a work should meet to be accepted for publication.
Due to the current difficulties in using this
indicator, others are used that can also offer
us an estimation of the quality of a thesis.
Among these, the following may be noted:
the total number of works derived from the
thesis (Np), the total number of citations of
works derived from the thesis (Nc), appointments for work or the ratio between Nc and
Np, number of significant works (Ns), number of citations of the most cited works (Cq),
number of works that have an amount of citations equal or superior to a stated number
(Index h of Hirsch). All these itemsallow us
to evaluate, in varying degrees, the quality of
a publication and can be used as much as the
derivatives of a thesis.
Finally, we wish to state that, while other
means of measuring the impact of a thesis in
its field of research exist, the most important
qualifier is its contribution to the construction
of knowledge that leads to intervention into
and resolution of problems, in our case educa-
tional ones. It is therefore unacceptable that
many theses are not intended for publication.
Unfortunately, a good part of what is called
gray literature in the Social Sciences and
Education consists of unpublished theses.
Currently, with the digital support and the
resources offered by the internet, the possibilities of disseminating theses and making
them accessible have greatly increased; even
though this situation has also led to other
problems (intellectual property, author’s
rights, etc.) that will need to be looked into
(Moralejo Alvarez, 2000; Merlo Vega y Sorli
Rojo, 2002; Orea, 2003).
5.- A model proposal to evaluate doctoral
theses.Once we have analyzed the different reports
involved in the evaluation of a doctoral thesis
we should advance the proposal for a model
that could constitute a framework for the
work to be carried out by the evaluators at
different times or stages of the process. To
this end we have developed a comprehensive
model that includes the main items to be
taken into account when reviewers make their
judgments about the doctoral theses. Our
purpose has been to set up a tool that can act,
in part, as a guide including the main items
that should be subject to analysis and assessment; and, secondly, a tool to capture and
contrast the quantitative and qualitative assessments that are made by different evaluators. In any case, we wish to consider our
model as an open proposal that each evaluator will have to adapt in each case.
For the development of this model, we have
used as a theoretical framework the literature
about the development of doctoral theses
(Alcina, 1994; Sierra, 1999; Blaxter et al,
2000, López Yepes et al, 2005, Uriz et al,
2006)), as well as some concrete proposals
recently written by other researchers from
similar fields (López Yepes et al 2008). To
develop the procedure we have used a simple
technique: pose some questions that the
evaluator should formulate about the basic
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 7
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
aspects that a thesis should comply with as a
scientific work, specifying in each case the
criteria of evaluation that we use to deliver
valuable judgments.
5.1 - Criteria for evaluating the report on the
thesis.When an evaluator is faced with a thesis report, the questions that should logically be
formulated in order to carry out its assessment are very simple and focus on four broad
dimensions or sections that should be specified precisely in this type of work: the statement of the problem, the specification of the
research methodology, the analysis of the
results, and the assessment of the report
submitted as a thesis. On each of these dimensions, an assessment should be made
adopting the evaluation criteria for scientific
articles themselves.
A.- The problem under investigation.1. Questions: What is the problem under investigation? Does it satisfy the
required characteristics to be considered a scientific investigation? Is it
formulated in an appropriate and clear
form? Are the hypothesis and objectives specified correctly? Is it original? Does it relate to the available
knowledge up to the present time?
Was a good review of the literature
made? Is it up to date? Does the fundamental theory have internal and external consistency? Can it be proved?
Is it pertinent to carry out this type of
investigation at this time and in this
context? Can it be included within the
doctoral program and the lines of
work of the research group? Is it relevant? Does it address an important
problem from a scientific point of
view?, etc.
2. Evaluation criteria: clarity, originality, pertinence, and relevance of the
problem.
B.- Research methodology used.1. Questions: What research methodology is used? ¿Is it consistent with the
objectives? Is it pertinent? Is it the
most appropriate at the present time
and in this context? Does it correctly
specify all of the steps? Does it question the use of alternative methods?
How were the samples selected? Do
the instruments used satisfy the necessary conditions? Are the analytical
tools handled adequately? Does it present new methods or techniques? Is
the methodology applied in a parsimonious form? Does it present all of
the necessary information to justify
the conclusions? Are adequate rates
of reliability and validity presented in
the results? Is the work replicable?,
etc.
2. Evaluation criteria: suitable methodology, accurate, parsimonious, and
replicable.
C.- Analysis of the Results.1. Questions: What results support the
thesis? Are they presented adequately? Are they justified from the
findings? Are they understandable? Is
there any kind of bias? Are they significant for the development of
knowledge? Are they novel? Do they
reaffirm or contradict previous
knowledge? Can they be useful? Are
they transferable? Are they presented
in a critical fashion? Does it propose
other outstanding issues for investigation? Appropriate for dissemination?
2. Evaluation criteria: understandable
results, significance, novelty, and usefulness.
D.- Formal aspects of the report.1. Questions: Does the thesis satisfy the
necessary conditions to be considered
a scientific work? Are the questions
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 8
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
presented in a clear fashion? Is it
thorough? Is it well written? Is the
style appropriate? Is the data included
easily understandable? Are the
sources specified? Is it well cited?
Does it include up-to-date references?
Does it conform to the standards of
scientific works?, etc.
2. Evaluation criteria: clarity of the report, precision of the content, adequate citation, and adjusted to scientific standards.
5.2.- Criteria for evaluating the presentation
and defense of the thesis.
As stated above, the examiners should evaluate the presentation and defense of the doctoral thesis at the viva. The questions and
criteria to be used for evaluation are:
1. Questions: Has the content of the
work been presented to the public in a
clear and understandable manner?
Has the candidate focused his or her
presentation on the fundamental aspects of the dissertation? Have the
technical devices contributed effectively to the presentation? Was the
candidate open to criticism, observations, and pertinent suggestions formulated during the presentation? Has
he/she answered all of the theoretical
and methodological questions posed
in relation to his/her work? Were the
responses adequate and correct?, etc.
2. Evaluation Criteria: Clarity in the
presentation of the work and adequacy of the defense given.
5.3.- Relative criteria about the impact of the
thesis.
Although we have already said that the
evaluation of the thesis’ scientific impact
should be evaluated over a period of time, at
the time of the defense, a set of indicators
could be adopted in order to advance prospective judgments about the contributions to
scientific knowledge that can be expected
from the work in question. Hence an initial
assessment of the impact of the thesis is in
order in these terms:
1. Questions: Has the work had any
kind of projection as of yet? Have
new lines and research projects been
generated from the thesis? Does it
provide methodologies of interest for
researchers? Impact of the work on
the community? Type of output publications? Presentations in conferences
and seminars? Invitations received?
Dissemination of the thesis among the
scientific community?
2. Evaluation criteria: Screening in the
field of research and impact of contributions in the scientific field.
Following from the above, we have developed a number of protocols (see the annex)
that present the practical dimensions and criteria which we believe the evaluators of the
thesis should take into account when giving
their personal assessment. The protocol
specifies the criteria that we have identified
as appropriate for the report prior to processing both by external evaluators and by the
members of the panel appointed for this purpose (I); those relating to the exposition and
defense of the thesis by the doctoral student
in the public session (II); and those that can
be used in a first assessment on the likely
impact of the work (III), although logically
the latter course should be disaggregated and
nuanced in terms of the purpose that justifies
the assessment post-thesis in each case. We
propose a double scaled evaluation postthesis- quantitative and qualitative- revolving
around the 20 established criteria we believe
have facilitated the formulation of the overall
thesis, the proposal to grant the qualification
and preparation of mandatory reports.
The protocol that is attached regarding the
evaluation of the report (IV) – especially the
final table- may estimulate the exchange of
views and dialog between the members of the
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 9
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
panel at the time of establishing a joint
evaluation, determining whether or not to
pass the thesis, and deciding an appropriate
grade. As stated in the regulations "the board
shall issue a report and the overall grade for
the thesis according to the following scale:
fail, pass, good and excellent. The board may
grant the distinction <cum Laudem> if the
overall mark is excellent and has been
reached unanimously" (art. 21.7). We believe that the use of scales as presented here
helps to introduce rational criteria in the
evaluation of the thesis, especially when the
time comes to decide the grade of the thesis.
Finally, we would like to stress that the
model we have presented constitutes an open
protocol for the scientific community that
may be subject to discussion and improvement by adding or changing certain aspects
and criteria. As with all personal work, we
are aware of its limitations, but we also believe that this may serve as an initial proposal for the construction of a useful and
practical tool in the evaluation of theses.
Employing an attached summary table may
contribute effectively to the evaluation of the
thesis. At any rate, it is the evaluators and
the members of the viva panels who will
judge whether the proposed criteria are appropriate to the extent that they facilitate
their task when it comes to making observations and judgments about the thesis, and
consequently, taking decisions on the appropriate grading.
References
Alcina, J. (1994). Aprender a investigar:
métodos de trabajo para la redacción de
tesis doctorales (Humanidades y Ciencias
Sociales). Madrid: Compañía Literaria.
Blaxter, L., Hughes, C. y Tight, M. (2000).
Cómo se hace una investigación. Barcelona:
Gedisa.
Carreras Panchón, A. (Coord.) (1994). Guía
práctica para la elaboración de un trabajo
científico. Bilbao: CITA, Publicaciones y
Documentación.
De Miguel, M. (1997). La evaluación de la
actividad investigadora del profesorado en
el ámbito de las ciencias de la educación.
Revista de Investigación Educativa, 15 (1),
171-186.
De Miguel, M. (2010). La evaluación y mejora de los estudios de doctorado. Revista de
Educación, 352 (en prensa).
Desantes, J. y López Yepes, J. (1996).
Teoría y técnica de la investigación
científica. Madrid: Síntesis.
García de la Fuente, O. (1994). Metodología
de la investigación científica. Madrid: Ediciones CEES.
López Yepes, J., Fernández Bajón, M.T. y
Prat, J. (2005). Las tesis doctorales: Producción, evaluación y defensa. Madrid:
Fragua.
López Yepes, J., Fernández Bajón, M.T.,
Orera, L., Sánchez Vigil, J.M., Martínez
Montalvo, E., Hernández Pacheco, F., Prat,
J. y Sanchez, C.M. (2008). Criterios para la
evaluación de tesis doctorales. Revista General de Información y Documentación, 18,
295-322.
Merlo Vega, J, y Sorli Rojo, A. (2002).
Bases de datos y recursos en Internet sobre
tesis doctorales. Revista española de documentación científica, 25 (1), 95-106.
Ministerio de Educación y Ciencia. Real
Decreto 1393/2007, de 29 de octubre, por el
que se establece la ordenación de las enseñanzas universitarias oficiales. (BOE, 30
de Octubre 2007, núm. 260, 44.037-44.048)
Moralejo Álvarez, M. (2000). Las tesis doctorales de las universidades españolas: control bibliográfico y acceso. Revista General
de Documentación e Información, 10 (1),
235-243.
Orea, L. (2003). La edición digital de tesis
doctorales: hacia la resolución de problemas
de accesibilidad. Revista interamericana de
Bibliotecología, 26 (1), 11-35.
Sierra Bravo, R. (1999). Tesis doctorales y
trabajos de investigación científica (5ª
edición). Madrid: Paraninfo.
Sánchez Rodríguez, E. (Coord). (2002).
Estudios de tercer ciclo en España. Programas de doctorado, tesis doctorales, indi-
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 10
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
cadores de calidad del tercer ciclo. MEC/
Dirección General de Universidades, Programa Estudios y Análisis.
Úriz, Mª J., Ballestero, A., Viscarret, J.J. y
Ursúa, N. (2006) Metodología para la investigación. Pamplona: Ediciones, Eunate.
Valcárcel, M. (Direct.) (2002). El doctorado
en las universidades españolas. Situación
actual y propuestas de mejora. Cordoba:
MEC/Dirección General de Universidades,
Programa Estudios y Análisis.
Notes
[1] Recently (in March 2010) the Spanish
Universities’ Secretary General, Ministry of
Education, circulated a draft of the Royal
Decree Project to regulate official Doctoral
Education. Although the text is not yet final,
the specific regulations regarding the evaluation and defence of theses do not introduce
substantial changes, with the exception of the
proposal suggesting that the distinction “cum
laude” not exceed, annually, 20% of the appointed and successful theses.
ANNEX 1
I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS
CRITERIA
Clarity of the
problem
Originality
A: The problem
under
investigation
Pertinence
Relevance
QUESTIONS
What is the problem under
research? Does it satisfy the
required characteristics to be
considered a scientific investigation? Is it formulated in an
appropriate and clear form?
Are the hypotheses and objectives specified correctly? Does
it have a good theoretical basis
or foundation?, etc.
Is it original? Which sources of
data and information are resorted to? Was a good review
of the literature made? Is it up
to date? Does it relate to the
current knowledge up to the
present time? Are new methods or procedures presented?,
etc.
Is it pertinent to carry out this
type of investigation at the
present time and in this context? Does it register along the
lines of the doctoral program
and/or the research group?, etc.
Is it relevant? Is it considered
an important problem to
resolve in the field of
education? Is it adequately
justified? Will it contribute
useful knowledge? Does it
respond to important current
demands?
11
2
3
4
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
ASSESSMENTS
pag. 11
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
B: Methods of
research used
I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS (continuation)
CRITERIA
QUESTIONS
1 2 3 4
ASSESSMENT
What research methods are
used? Does a connection between the theoretical framework
and the methodology exist? Is it
Adecuate
adecuately justified? Is it writmethodology
ten in the most coherent way for
what it is trying to say? Does it
introduce some new contribution? , etc.
Are the design, the variables,
the samples, the procedures for
collecting data, and the data
analysis techniques presented in
a detailed and sufficient
Correct specification
manner? Are all the steps
specified in correct fashion? Do
the instruments have reasonable
levels of reliabilty and validity?,
etc.
Is the methodology applied in a
parsimonious form? Are the
investigative
procedures
Parsimonious
specified step by step? Is the
application
process of data analysis
thorough? Does it present all of
the necessary data to justify the
conclusions?, etc.
Is it replicable? Can the design
and procedure that were used be
replicated with other projects?
Replicable method
Are there other works that exist
that can be used as a reply? Are
these works cited?, etc.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 12
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
C: Analysis of
the
Results
I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS (continuation)
CRITERIA
QUESTIONS
1 2 3 4
ASSESSMENT
What results and conclusions support the thesis? Are they presented
in an adecuate fashion? Are they
understandable? Are they warUnderstandable
ranted from the obtained data? Do
results
they answer the proposed hypotheses and objectives? Is there a disagreement with the conclusions?,
etc.
Are they important for the development of knowledge? Do they
reaffirm or contradict previous
Significance
knowledge? Do they contribute to
the resolution of specific problems?, etc.
.
From a theoretical and methodological point of view, are they
novel? Are they presented in a
Originality
critical fashion? What are the limitations of the study? Does it propose other outstanding issues for
investigation?, etc.
Can they be useful? For whom will
they be useful? Does it present any
guarantee in this regard? Should it
Usefulness
be published? Does it carry out
proposals and recommendations
useful for immediate application?,
etc.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 13
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
D: Formal
Aspects of the
Report
E: Evaluation
of the Presentation and Public
Defense
I.-PROTOCOL FOR EVALUATING THE REPORT OF THE THESIS (continuation)
CRITERIA
QUESTIONS
1 2 3 4 ASSESSMENT
Are the contents presented in a
clear form? Is the report well
written? Is the style appropriate?
Clarity
Does it have an adequate structure? Are the data, the tables,
and the graphs that are included
easily understandable?
Is there internal consistency
between the theory and the pracPrecision
tice? Is the report thorough? Is it
elaborated in a systematic form?
Is scientific language used?, etc.
Is it well cited? Does it specify
the sources used? Does it include
up-to-date references? Are the
Documentation
quotations and references pertinent? Do they match the text?,
etc.
Does the report have an adequately formal presentation?
Does the thesis satisfy the necesAdjusted to
sary conditions to be considered
scientific standards
a scientific work? Does the writing conform to the norms of
scientific works? Is it publishable?, etc.
II.- PROTOCOL FOR EVALUATING THE PRESENTATION AND DEFENSE OF THE
THESIS
CRITERIA
QUESTIONS
1 2 3 4
ASSESSMENT
Has the content of the work been
presented to the public in a clear
and understandable manner? Have
Clarity of the
the fundamental aspects of the
presentation
work been stressed?
Did the technical means used
contribute effectively to the presentation that was made?, etc.
Has the doctoral candidate expressed openness to criticism and
relevant comments made by the
Adequacy of the
panel? Has he/she answered all of
defense
the theoretical and methodological
questions posed in relation to
his/her work? Were the responses
adequate and correct?, etc.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 14
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
F: Evaluation
of the Impact
III.- PROTOCOL FOR EVALUATING THE IMPACT OF THE THESIS
CRITERIA
QUESTIONS
1 2 3 4
ASSESSMENT
Are new lines of research on the
problem under investigation proposed? Does it contribute interesting methodologies for researchers? Has the thesis had an impact
Screening in the
field of research
in other projects or investigations? Have new valuable models
or tools been generated to be
implemented in other projects?,
etc.
Is there any sort of dissemination
of the work at a scientific level:
publication, seminars, communiContributions to
cations, etc.? Is there any expectascientific
tion of publishing? Has any kind
knowledge
of model or patent been generated? What impact can be expected from the contributions
advanced by the thesis?, etc.
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 15
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
SUMMARY
IV. PROTOCOL FOR THE REPORTS ON THE THESIS
INFORMATION
ABOUT THE
DISSERTATION
Title
Author
Supervisor
Department
University
INFORMATION
ABOUT THE
EVALUATOR
Name
Institution
Appointment 3
DIMENSIONS TO BE
EVALUATE
SCORE
ASSESSMENT
A: The problem / hypothesis under investigation
B: Methodology of the
research used
C: Analysis of the Results
ASSESSMENT OF
THE
THESIS
D: Formal aspects of the
report
E: Evaluation of the
presentation and public
defense
F: Evaluation of the Impact
OVERALL GRADE
OBSERVATIONS AND COMMENTS:
________________________________________________________________________________________________
________________________________________________________________________________________________
________________________________________________________________________________________________
Date and Signature:
ANNEX NOTES
[2] 1= Fail
2= Pass
3= Good
4= Excellent
[3] Indicate whether the Doctoral Thesis is presented from within or outside the Department/University. Whether you
are a member of the proposed Panel, and in that case, the Panel position you occupy
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 16
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1, p. 1-17.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
ABOUT THE AUTHORS / SOBRE LOS AUTORES
de-Miguel, Mario (mario@uniovi.es): Chair of Research Methods in Education at the University of Oviedo
(Spain). It is integrated in the scientific area of 'Research Methods and Diagnostics in Education' (MIDE).
His mailing address is: Departamento de Ciencias de la Educación. C/ Aniceto Sella s/n. 33005-Oviedo
(España). Find other articles by this author in Scholar Google
ARTICLE RECORD / FICHA DEL ARTÍCULO
Reference /
Referencia
Title / Título
Authors /
Autores
Review /
Revista
ISSN
Publication
date /
Fecha de publicación
Abstract /
Resumen
Keywords /
Descriptores
de-Miguel, Mario (2010). The evaluation of doctoral thesis. A model proposal. RELIEVE, v. 16, n. 1.
http://www.uv.es/RELIEVE/v16n1/RELIEVEv16n1_4.htm
The evaluation of doctoral thesis. A model proposal. [La evaluación de tesis doctorales. Propuesta de un
modelo].
de-Miguel, Mario
RELIEVE (Revista ELectrónica de Investigación y EValuación Educativa / E-Journal of Educational Research,
Assessment and Evaluation), v. 16, n. 1.
1134-4032
2009 (Reception Date: 2009 September 16; Approval Date: 2010 April 20; Publication Date: 2010 April 20).
The aim of this work is the development of a model that can be used as a framework for evaluators and PhD tribunal members, either during the previous administrative processing phase or at the actual exposition and viva
on doctoral dissertations. To this end, once reviewed the existing literature and normative, a new proposal is laid
out on the main aspects and criteria to be analysed and assessed from any given scientific work, thus introducing
a tool that facilitates and specifies this work. Our model is intended to be an open proposal on which we would
welcome any contribution and suggestion.
El objetivo del presente trabajo es la elaboración de un modelo que pueda ser utilizado como marco de referencia
por los evaluadores y miembros de tribunales de las tesis doctorales tanto en la fase previa a la tramitación administrativa como en el acto de exposición y defensa de la misma. Para ello, una vez revisada la literatura y la
normativa vigente al respecto, efectuamos una propuesta sobre los principales aspectos y criterios que deben ser
objeto de análisis y valoración en todo trabajo científico, y presentamos una herramienta que facilita y precisa
esta tarea. Nuestro modelo constituye una propuesta abierta sobre la que esperamos aportaciones y sugerencias.
Doctoral thesis, Doctoral dissertations evaluation, doctoral thesis quality, doctoral dissertations evaluation
model.
Tesis doctoral, evaluación de tesis doctorales, calidad de las tesis doctorales, modelo de evaluación de las tesis
doctorales
Institution /
Departamento de Ciencias de la Educación, Universidad de Oviedo - (España).
Institución
Publication site
http://www.uv.es/RELIEVE
/ Dirección
Language /
English and Spanish versions (Title, abstract and keywords in Spanish and English)
Idioma
RELIEVE
Revista ELectrónica de Investigación y EValuación Educativa
E-Journal of Educational Research, Assessment and Evaluation
[ISSN: 1134-4032]
© Copyright, RELIEVE. Reproduction and distribution of this articles it is authorized if the content is no modified
and their origin is indicated (RELIEVE Journal, volume, number and electronic address of the document).
© Copyright, RELIEVE. Se autoriza la reproducción y distribución de este artículo siempre que no se modifique el
contenido y se indique su origen (RELIEVE, volumen, número y dirección electrónica del documento).
Revista ELectrónica de Investigación y EValuación Educativa [ www.uv.es/RELIEVE ]
pag. 17