Kernel and Probit Estimates in Quantal Bioassay

Abstract

A kernel method for the estimation of quantal dose-response curves is considered. In contrast to parametric modeling, this local smoothing method does not require any assumptions beyond smoothness of the dose-response curve and, in this sense, is nonparametric. In finite-sample situations, the kernel estimate of the dose-response curve is not necessarily monotone and, therefore, if the additional assumption of monotonicity of the dose-response curve is made, a monotonized version is discussed. Bias, variance, asymptotic normality, and uniform consistency, including rates of convergence of kernel estimates, are derived and applied to establish consistency and limiting distribution of kernel estimates of the EDα. Here, EDα is the effective dose at level α, that is, the dose where 100α% of the subjects show a response. The properties of the kernel estimated EDα are compared with the corresponding properties of the maximum likelihood estimator assuming the probit model. Practical application of the kernel method requires choice of a kernel function and of a bandwidth (smoothing window), and methods for global as well as local bandwidth selection are discussed. Another point of interest is the construction of confidence intervals for the estimated EDα. For the kernel method, two asymptotically consistent approaches are presented. These are compared in a simulation study, where the behavior of kernel and probit estimates of the EDα, α = .01, .05, .1, and .5, and of corresponding confidence intervals is observed for four different models (two probit models, one Weibull model, and one normal mixture model). The choice of different kernels and of local bandwidths was compared in another Monte Carlo study. The observed finite sample behavior of the kernel estimate of the ED50 was considerably better (gain of 40%-70% in terms of mean squared error) than that of the probit method under the two nonsigmoid models, whereas it was not drastically worse under the probit models (loss of 20%-30% in terms of mean squared error). Application of higher-order kernels and local bandwidth choice turned out to yield favorable versions of the kernel method. The results concerning observed coverage probabilities for the confidence intervals based on different approaches were more mixed.

Key Words:

• Asymptotic distribution
• Confidence interval
• Curve estimation
• Dose-response curve
• Effective dose 50
• Monte Carlo study