<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>차라투 블로그</title>
<link>https://blog.zarathu.com/</link>
<atom:link href="https://blog.zarathu.com/index.xml" rel="self" type="application/rss+xml"/>
<description>차라투의 소식, 인사이트 및 심층 사례 연구</description>
<generator>quarto-1.8.26</generator>
<lastBuildDate>Fri, 02 Jan 2026 00:00:00 GMT</lastBuildDate>
<item>
  <title>비정규 분포의 데이터에 GEE를 사용할 수 있을까?</title>
  <dc:creator>Sangho Oh</dc:creator>
  <link>https://blog.zarathu.com/posts/2026-01-02-LMM, GEE/</link>
  <description><![CDATA[ 
<!-- Google tag (gtag.js) -->
<script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script>





<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>의료 분야의 데이터는 같은 환자에 대해 반복 측정하거나, 같은 병원에서 측정하는 자료 등으로 구성되어있는 경우가 많다. 즉, 데이터들 간에 상관관계가 존재하기 때문에 독립성 가정이 깨져, 회귀분석을 사용하지 못한다. 그렇기 때문에 의료 데이터 분석에는 LMM(Linear Mixed model)과 GEE(Generaluzed Estimating Equation)이 자주 사용된다. 이러한 방법들을 사용할 때, 데이터의 형태에 따라 제약조건들이 존재하고, 연구자들에게 혼동을 주기도 한다. 본 Article에서는 LMM의 제약조건으로 인해, GEE의 사용가능여부를 혼동하게 되는 상황에 대해서 알아볼 것이다.</p>
</section>
<section id="lmm의-기본-구조" class="level2">
<h2 class="anchored" data-anchor-id="lmm의-기본-구조">LMM의 기본 구조</h2>
<p>LMM은 전체 환자 집단에 공통적으로 적용되는 평균적인 효과 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">와 환자 개개인의 특성을 반영하는 확률 변수 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">를 하나의 선형 결합으로 표현한다. <img src="https://latex.codecogs.com/png.latex?m">명의 환자(클러스터)와 각 환자 <img src="https://latex.codecogs.com/png.latex?i">에 대한 <img src="https://latex.codecogs.com/png.latex?n_i">번의 반복 측정이 있을 때, 모델은 다음과 같이 정의된다.</p>
<p>LMM의 기본적인 구조는 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D_i%20=%20%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D%20+%20%5Cmathbf%7BZ%7D_i%5Cmathbf%7Bb%7D_i%20+%20%5Cboldsymbol%7B%5Cepsilon%7D_i,%20%5Cquad%20i=1,%5Cdots,m%0A"> 여기서:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7By%7D_i">: <img src="https://latex.codecogs.com/png.latex?i">번째 환자의 반응변수 벡터 (<img src="https://latex.codecogs.com/png.latex?n_i%20%5Ctimes%201">)</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D_i">: 고정 효과에 대한 설계 행렬 (<img src="https://latex.codecogs.com/png.latex?n_i%20%5Ctimes%20p">)</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">: 고정 효과 회귀 계수 벡터 (<img src="https://latex.codecogs.com/png.latex?p%20%5Ctimes%201">)</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BZ%7D_i">: 랜덤 효과에 대한 설계 행렬 (<img src="https://latex.codecogs.com/png.latex?n_i%20%5Ctimes%20q">)</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">: 랜덤 효과 벡터 (<img src="https://latex.codecogs.com/png.latex?q%20%5Ctimes%201">)</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cepsilon%7D_i">: 잔차 벡터 (<img src="https://latex.codecogs.com/png.latex?n_i%20%5Ctimes%201">)</li>
</ul>
<p>LMM의 핵심은 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">와 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cepsilon%7D_i">에 부여되는 확률 분포 가정에 있다. 가우시안 LMM은 이 두 요소가 서로 독립이며 다변량 정규분포를 따른다고 가정한다</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7Bb%7D_i%20%5C%5C%20%5Cboldsymbol%7B%5Cepsilon%7D_i%20%5Cend%7Bpmatrix%7D%20%5Csim%20%5Cmathcal%7BN%7D%20%5Cleft(%20%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7B0%7D%20%5C%5C%20%5Cmathbf%7B0%7D%20%5Cend%7Bpmatrix%7D,%20%5Cbegin%7Bpmatrix%7D%20%5Cmathbf%7BG%7D%20&amp;%20%5Cmathbf%7B0%7D%20%5C%5C%20%5Cmathbf%7B0%7D%20&amp;%20%5Cmathbf%7BR%7D_i%20%5Cend%7Bpmatrix%7D%20%5Cright)%0A"> 여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D">는 환자 간의 이질성(Between-subject variability)을 설명하는 공분산 행렬이며, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i">는 환자 내 잔차의 공분산 행렬이다. 가장 단순한 형태인 조건부 독립(Conditional Independence) 가정 하에서는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i%20=%20%5Csigma%5E2%20%5Cmathbf%7BI%7D_%7Bn_i%7D">가 된다. 이는 “환자 고유의 특성 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">가 주어졌을 때, 반복 측정된 오차들은 서로 독립적이다”라는 가정을 의미한다.</p>
</section>
<section id="lmm에서-조건부-분포와-marginal-분포" class="level2">
<h2 class="anchored" data-anchor-id="lmm에서-조건부-분포와-marginal-분포">LMM에서 조건부 분포와 Marginal 분포</h2>
<p>조건부 분포는 아래와 같이 얻어진다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D_i%5Cmid%20%5Cmathbf%7Bb%7D_i%20%5Csim%20%5Cmathcal%7BN%7D%5C!%5Cleft(%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D+%5Cmathbf%7BZ%7D_i%5Cmathbf%7Bb%7D_i,%5C%20%5Csigma%5E2%5Cmathbf%7BI%7D_%7Bn_i%7D%5Cright).%0A"> 이때, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">를 적분해 마지널(주변) 분포를 얻으면</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D_i%20%5Csim%20%5Cmathcal%7BN%7D(%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D,%20%5Cmathbf%7BV%7D_i)%0A"> 여기서 주변 공분산 행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">는 다음과 같이 유도된다. <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BV%7D_i%20=%20%5Cmathbf%7BZ%7D_i%20%5Cmathbf%7BG%7D%20%5Cmathbf%7BZ%7D_i%5E%5Ctop%20+%20%5Cmathbf%7BR%7D_i%0A"> 이 수식은 LMM이 데이터의 상관성을 어떻게 모델링하는지 명확히 보여준다. <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">는 랜덤 효과에 의한 변동(<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BZ%7D_i%20%5Cmathbf%7BG%7D%20%5Cmathbf%7BZ%7D_i%5E%5Ctop">)과 순수 오차에 의한 변동(<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i">)의 합으로 구성된다. 즉, LMM은 상관성을 구조적인 분산 성분(Variance Components) 의 결합으로 설명한다.</p>
<p>즉 LMM은 평균 구조 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D">와 함께, 반복측정 자료의 상관을 공분산 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">로 모델링한다.</p>
</section>
<section id="lmm의-정규성-가정이-혼동을-만드는-지점" class="level2">
<h2 class="anchored" data-anchor-id="lmm의-정규성-가정이-혼동을-만드는-지점">LMM의 “정규성 가정”이 혼동을 만드는 지점</h2>
<p>LMM의 파라미터 추정은 주변 우도(Marginal Likelihood) 함수를 최대화하는 과정이다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D_i%20%5Csim%20%5Cmathcal%7BN%7D(%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D,%5Cmathbf%7BV%7D_i(%5Cboldsymbol%7B%5Ctheta%7D))%0A"></p>
<p>이며, Log-likelihood는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cell(%5Cboldsymbol%7B%5Cbeta%7D,%5Cboldsymbol%7B%5Ctheta%7D)%0A=%0A-%5Cfrac%7B1%7D%7B2%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%0A%5Cleft%5B%0A%5Clog%5Cleft%7C%5Cmathbf%7BV%7D_i(%5Cboldsymbol%7B%5Ctheta%7D)%5Cright%7C%0A+%0A(%5Cmathbf%7By%7D_i-%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D)%5E%7B%5Ctop%7D%0A%5Cmathbf%7BV%7D_i(%5Cboldsymbol%7B%5Ctheta%7D)%5E%7B-1%7D%0A(%5Cmathbf%7By%7D_i-%5Cmathbf%7BX%7D_i%5Cboldsymbol%7B%5Cbeta%7D)%0A%5Cright%5D%0A+%20%5Ctext%7Bconst%7D.%0A"> 이다.</p>
<p>이 목적 함수는 데이터 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7By%7D_i">가 다변량 정규분포를 따른다는 전제하에 도출되었다. 그러나 실제 의료 데이터는 정규성을 심각하게 위배하는 경우가 빈번하다.</p>
<p>그러한 경우에 속하는 대표적인 데이터들은 아래와 같다.</p>
<ul>
<li>비연속적 결과 변수 : 치료 성공 여부(Binary), 발작 횟수(Count), 질병의 단계(Ordinal) 등은 정규분포로 근사하기 어렵다.</li>
<li>유계 데이터(Bounded Data): 통증 점수(VAS 0-10), 삶의 질 지수(0-1), 검사 수치(항상 양수) 등은 정의역이 제한되어 있어 정규분포의 무한한 지지 집합(Support) 가정과 충돌한다.</li>
<li>Skewness &amp; Kurtosis: 의료 비용이나 재원 기간 데이터는 전형적으로 오른쪽으로 긴 꼬리를 가진 분포(Log-normal or Gamma-like)를 보인다.</li>
</ul>
<p>이러한 비정규 데이터에 LMM을 강제로 적용할 경우 발생할 수 있는 문제는 단순히 모델 적합도가 떨어지는 것에 그치지 않는다. 다음과 같은 심각한 문제상황을 발생 시킬 수 있기 때문이다.</p>
<p><strong>-표준오차의 왜곡</strong>: 정규성 가정에 기반한 분산 추정량은 데이터의 이분산성(Heteroscedasticity)이나 비정규성을 반영하지 못해, 표준오차를 부정확하게 계산한다. 이는 신뢰구간의 포함 확률을 떨어뜨리고 p-value의 신뢰성을 훼손한다.</p>
<p><strong>-추정 효율성 저하</strong>: MLE는 정규분포 하에서만 효율적이다. 분포가 다를 경우, 더 적절한 분포를 가정한 모델보다 추정량의 분산이 커질 수 있다.</p>
</section>
<section id="gee의-기본-구조-추정방정식-quasi평균분산-상관구조" class="level2">
<h2 class="anchored" data-anchor-id="gee의-기본-구조-추정방정식-quasi평균분산-상관구조">GEE의 기본 구조: 추정방정식 + quasi(평균–분산) + 상관구조</h2>
<p>GEE(Generalized Estimating Equations)는 특정 분포의 우도(likelihood)를 세우기보다, 반복측정 자료에서 평균모형을 기반으로 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">를 추정하는 <strong>추정방정식(estimating equation)</strong> 접근이다. 즉, 데이터의 전체 결합 분포(Joint Distribution)를 완벽하게 명시하지 않더라도, 관심 있는 평균 파라미터(<img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">)를 일관되게 추정할 수 있다는 점에 착안한다.</p>
</section>
<section id="평균모형과-추정방정식" class="level2">
<h2 class="anchored" data-anchor-id="평균모형과-추정방정식">평균모형과 추정방정식</h2>
<p>평균모형은 다음과 같이 둔다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ag(%5Cmu_%7Bij%7D)=%5Ceta_%7Bij%7D=%5Cmathbf%7Bx%7D_%7Bij%7D%5E%7B%5Ctop%7D%5Cboldsymbol%7B%5Cbeta%7D.%0A"></p>
<p>환자 <img src="https://latex.codecogs.com/png.latex?i"> 단위로</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BY%7D_i=(Y_%7Bi1%7D,%5Cdots,Y_%7Bin_i%7D)%5E%5Ctop,%5Cqquad%0A%5Cboldsymbol%7B%5Cmu%7D_i=(%5Cmu_%7Bi1%7D,%5Cdots,%5Cmu_%7Bin_i%7D)%5E%5Ctop,%0A%5Cqquad%0A%5Cmathbf%7BD%7D_i=%5Cfrac%7B%5Cpartial%20%5Cboldsymbol%7B%5Cmu%7D_i%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%5E%5Ctop%7D%0A"></p>
<p>를 정의하면, GEE는 다음 방정식을 만족하는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">를 구한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BU%7D(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Csum_%7Bi=1%7D%5E%7Bm%7D%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20(%5Cmathbf%7By%7D_i%20-%20%5Cboldsymbol%7B%5Cmu%7D_i)%20=%20%5Cmathbf%7B0%7D%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">는 환자 <img src="https://latex.codecogs.com/png.latex?i"> 내부의 공분산을 나타내며, GEE에서는 이를 quasi(평균–분산)와 상관구조를 조합해 구성한다. 이는 정규분포를 따르는 독립데이터에 대해 스코어 함수를 0으로 놓는 방정식의 해를 구하는 MLE의 방식을 상관된 데이터로 확장시킨 일반화 추정 방정식이다. (이 식의 구조는 가중 최소 제곱법(Weighted Least Squares)과 유사하다. 잔차 <img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7By%7D_i%20-%20%5Cboldsymbol%7B%5Cmu%7D_i)">에 공분산의 역행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i%5E%7B-1%7D">을 가중치로 곱함으로써, 변동이 크거나 상관성이 높은 데이터 포인트의 영향력을 적절히 조절한다.)</p>
</section>
<section id="quasi평균분산-관계-mathbfv_i를-만들기-위한-2차-구조" class="level2">
<h2 class="anchored" data-anchor-id="quasi평균분산-관계-mathbfv_i를-만들기-위한-2차-구조">Quasi(평균–분산 관계): <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">를 만들기 위한 2차 구조</h2>
<p>GEE가 “분포를 가정하지 않는다”고 할 때, 이는 확률 밀도 함수 전체를 특정하지 않는다는 의미이다. 대신 GEE는 첫 번째 모멘트(평균)와 두 번째 모멘트(분산)의 관계만을 정의하는 준우도(Quasi-Likelihood) 접근을 취한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbb%7BE%7D(Y_%7Bij%7D)=%5Cmu_%7Bij%7D,%5Cqquad%0A%5Cmathrm%7BVar%7D(Y_%7Bij%7D)=%5Cphi%5C,V(%5Cmu_%7Bij%7D).%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?V(%5Cmu)">는 분산함수, <img src="https://latex.codecogs.com/png.latex?%5Cphi">는 scale(분산 크기)이다. 즉 GEE에서 필요한 것은 <img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D(Y_%7Bij%7D%5Cmid%20%5Cmathbf%7Bx%7D_%7Bij%7D)">가 어떻게 변하는지(평균모형)와, 그 주변 변동을 <img src="https://latex.codecogs.com/png.latex?%5Cphi%20V(%5Cmu_%7Bij%7D)">로 어떻게 요약할지(분산함수)이다.</p>
<p><img src="https://latex.codecogs.com/png.latex?V(%5Cmu)">는 “분산이 평균에 따라 어떻게 달라지는지”를 정하는 선택이다. 흔히 사용되는 설정은 다음과 같다.</p>
<ul>
<li>Gaussian-like: <img src="https://latex.codecogs.com/png.latex?V(%5Cmu)%20=%201"> (분산이 일정함)</li>
<li>Bernoulli-like: <img src="https://latex.codecogs.com/png.latex?V(%5Cmu)%20=%20%5Cmu(1-%5Cmu)"> (평균이 0.5일 때 분산 최대)</li>
<li>Poisson-like: <img src="https://latex.codecogs.com/png.latex?V(%5Cmu)%20=%20%5Cmu"> (평균과 분산이 같음)</li>
<li>Negative Binomial-like: <img src="https://latex.codecogs.com/png.latex?V(%5Cmu)%20=%20%5Cmu%20+%20k%5Cmu%5E2"></li>
</ul>
<p>이러한 설정은 데이터 생성 분포를 완벽하게 묘사하려는 것이 아니라, 회귀 계수 추정에 필요한 최소한의 정보(평균-분산 관계)만을 모델링하는 것이다. 따라서 <strong>실제 데이터가 해당 분포를 정확히 따르지 않더라도, 평균 모델이 올바르다면 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">는 참값으로 수렴하는 일치성(Consistency)을 유지한다.</strong></p>
</section>
<section id="gee의-핵심-작업-공분산-행렬과-샌드위치-추정량" class="level2">
<h2 class="anchored" data-anchor-id="gee의-핵심-작업-공분산-행렬과-샌드위치-추정량">GEE의 핵심: 작업 공분산 행렬과 샌드위치 추정량</h2>
<p>GEE의 가장 독창적인 부분은 반복측정 자료에서 한 개인(클러스터) 내부의 상관을 “완벽히 맞추려고” 하기보다, <img src="https://latex.codecogs.com/png.latex?%5Cbeta">의 일관된 추정에 필요한 수준으로 공분산 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">를 분해하고 처리하는 방식에 있다. <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">는 다음과 같이 분해된다. <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i%20=%20%5Cphi%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D%20%5Cmathbf%7BR%7D_i(%5Cboldsymbol%7B%5Calpha%7D)%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D"> 여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D_i">는 각 시점의 분산 크기(scale)를 담당하는 대각행렬로, 대각 원소가 분산함수 <img src="https://latex.codecogs.com/png.latex?V(%5Cmu_%7Bij%7D)">로 주어진다. 반면 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Cboldsymbol%7B%5Calpha%7D)">는 시점 간의 상관 형태(패턴)만을 따로 모아 둔 행렬이다. 이때 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D_i%5E%7B1/2%7D">가 각 시점의 표준편차 역할을 하므로, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D_i%5E%7B1/2%7D%5Cmathbf%7BR%7D_i(%5Cboldsymbol%7B%5Calpha%7D)%5Cmathbf%7BA%7D_i%5E%7B1/2%7D">는 표준편차와 상관구조를 결합해 공분산을 재구성하는 전형적인 형태가 된다.</p>
<p>따라서 위 분해는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D_i">: 각 시점의 분산 크기(marginal variance), <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Cboldsymbol%7B%5Calpha%7D)">: 시점 간 상관의 패턴(correlation pattern), <img src="https://latex.codecogs.com/png.latex?%5Cphi">: 전체 분산 스케일(scale) 을 분리해서 다루겠다는 의미이다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Cboldsymbol%7B%5Calpha%7D)">을 “<strong>작업 상관 행렬 (Working Correlation Matrix)</strong>”이라 부르는 이유는 연구자가 선택한 상관구조가 실제 데이터의 진짜 상관구조와 같을 필요가 없고, 실제로 다를 수 있음을 강조하기 위해서이다. 예를 들어 exchangeable, AR(1), independence 등은 분석을 진행하기 위한 합리적 근사일 뿐이며, 참모형을 선언하는 것이 아니다. GEE의 핵심 정리는 작업 상관구조 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i">를 잘못 설정하더라도(working misspecification), 평균모형이 올바르게 주어졌다면 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">는 여전히 일치 추정량(consistency)으로 남는다는 것이다. 다만 이 경우 모형을 그대로 믿고 계산한 표준오차(model-based/naive SE)는 왜곡될 수 있으므로, 강건한 표준오차(robust SE)를 사용해야 유효한 추론이 가능하다. 이 강건성을 실제로 보장하는 것이 <strong>샌드위치 공분산 추정량(sandwich covariance estimator)</strong>이며, GEE에서 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">의 공분산은 다음 형태로 추정된다. <img src="https://latex.codecogs.com/png.latex?%5Ctext%7BCov%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20%5Cmathbf%7BM%7D_0%5E%7B-1%7D%20%5Cmathbf%7BM%7D_1%20%5Cmathbf%7BM%7D_0%5E%7B-1%7D"> 여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BM%7D_0">는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BM%7D_0=%5Csum%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20%5Cmathbf%7BD%7D_i"> 로 정의되며, 작업 공분산 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">를 기준으로 한 “모델 기반 정보행렬”에 해당한다. 반면 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BM%7D_1">은 잔차의 경험적 변동성을 반영하는 항으로, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BM%7D_1=%5Csum%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20(%5Cmathbf%7By%7D_i-%5Cboldsymbol%7B%5Cmu%7D_i)(%5Cmathbf%7By%7D_i%20-%20%5Cboldsymbol%7B%5Cmu%7D_i)%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D"> 로 주어진다. 핵심은 가운데의 잔차 외적 <img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7By%7D_i-%5Cboldsymbol%7B%5Cmu%7D_i)(%5Cmathbf%7By%7D_i%20-%20%5Cboldsymbol%7B%5Cmu%7D_i)%5E%5Ctop">이 데이터가 실제로 보여주는 변동성(상관 포함)을 담고 있다는 점이다. 즉, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BM%7D_0">는 “우리가 가정한 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i"> 아래에서의 이론적 정보”를 제공하고, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BM%7D_1">은 “현실 데이터 잔차가 만들어내는 경험적 변동성”으로 작업 상관구조 오지정에서 생길 수 있는 표준오차 왜곡을 보정한다. 이 때문에 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i">를 완벽히 맞추지 못하더라도, 샌드위치 추정량을 사용하면 강건한 표준오차를 통해 유효한 통계적 추론이 가능해진다.</p>
</section>
<section id="비정규-outcome에서도-gee를-배제할-근거는-없다" class="level2">
<h2 class="anchored" data-anchor-id="비정규-outcome에서도-gee를-배제할-근거는-없다">비정규 outcome에서도 GEE를 배제할 근거는 없다</h2>
<p>LMM은 <strong>정규-우도 기반 추론</strong>이 기본이기 때문에, outcome이 정규에서 크게 벗어날 때 p-value/CI 해석이 민감해질 수 있다. 반면 GEE는 <strong>분포를 특정하지 않고(quasi로 평균–분산만 둠)</strong>, 환자 내 상관을 working 상관구조로 반영한 뒤, 표준오차는 robust(sandwich)로 정리한다. 따라서 <strong>outcome이 정규분포를 따르지 않는다는 이유만으로 GEE를 배제할 근거는 없다.</strong></p>
<p>다만 GEE가 무가정이라는 뜻은 아니다. 클러스터 간 독립(대개 환자 간 독립), 평균모형의 적절성, 그리고 클러스터 수 <img src="https://latex.codecogs.com/png.latex?m">이 충분하지 않을 때 robust 표준오차가 불안정할 수 있다는 점은 함께 고려해야 한다.</p>
</section>
<section id="상관구조-mathbfr_ialpha-independence-vs-exchangeable를-어떻게-이해하고-선택할까" class="level2">
<h2 class="anchored" data-anchor-id="상관구조-mathbfr_ialpha-independence-vs-exchangeable를-어떻게-이해하고-선택할까">상관구조 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">: independence vs exchangeable를 어떻게 이해하고 선택할까</h2>
<p>GEE에서 작업 상관행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Cboldsymbol%7B%5Calpha%7D)">는 환자 <img src="https://latex.codecogs.com/png.latex?i">의 반복측정 값들 사이의 상관을 반영하기 위한 장치다. 다만 핵심은 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i">가 “현실의 상관구조를 정확히 재현하는 참모형”이 아니라, 공분산 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">를 구성할 때 필요한 상관 패턴을 단순화해서 두는 working 가정이라는 점이다. 따라서 상관구조 선택을 “정답 고르기” 문제로 만들기보다는, 상관구조를 바꿔도 핵심 결론(예: <img src="https://latex.codecogs.com/png.latex?Time%20%5Ctimes%20Group"> 또는 slope 차이)이 유지되는지를 함께 보여주는 방식이 가장 설득력 있다.</p>
</section>
<section id="independence" class="level2">
<h2 class="anchored" data-anchor-id="independence">Independence</h2>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BR%7D_i(%5Calpha)=%5Cmathbf%7BI%7D_%7Bn_i%7D.%0A"></p>
<p>independence는 환자 내 상관을 0으로 두는 가정이다. 반복측정 자료에서 상관이 실제로 0인 경우는 흔하지 않지만, 이 선택은 종종 “상관을 무시한다”기보다 <strong>상관에 대한 가정을 최소화</strong>한다는 의미로 사용된다. 특히 GEE에서는 작업 상관구조가 틀려도 robust(Sandwich) 표준오차를 사용하면 추론이 가능하므로, independence는 충분히 정당한 기본 설정이 될 수 있다.</p>
<ul>
<li>장점: 구조가 단순하여 적합이 안정적이고, 관측 시점이 불규칙하거나 측정 횟수가 제각각인 불균형자료에서도 부담이 적다.</li>
<li>단점: 실제 양의 상관이 존재할 때는 추정 효율이 떨어질 수 있어(표준오차가 커져) 검정력이 낮아지는 방향으로 작용할 수 있다.</li>
</ul>
<p>즉 independence는 기준점으로 두고, 다른 상관구조와 비교했을 때 결론이 크게 달라지는지 확인하는 용도로도 자주 쓰인다.</p>
</section>
<section id="exchangeable-compound-symmetry" class="level2">
<h2 class="anchored" data-anchor-id="exchangeable-compound-symmetry">Exchangeable (compound symmetry)</h2>
<p><img src="https://latex.codecogs.com/png.latex?%0A(%5Cmathbf%7BR%7D_i)_%7Bjk%7D=%0A%5Cbegin%7Bcases%7D%0A1,&amp;%20j=k,%5C%5C%0A%5Crho,&amp;%20j%5Cneq%20k.%0A%5Cend%7Bcases%7D%0A"></p>
<p>exchangeable은 같은 환자 내 임의의 두 시점 관측치가 동일한 상관 <img src="https://latex.codecogs.com/png.latex?%5Crho">를 가진다고 가정한다. 실제로는 시간 간격이 멀어질수록 상관이 약해지는 경우가 많지만, 측정 횟수가 많지 않거나 분석의 초점이 “시간에 따른 평균 변화”에 있을 때는, 상관을 어느 정도 반영하면서도 단순한 구조라는 이유로 가장 흔히 선택된다.</p>
<ul>
<li>장점: 환자 내 상관을 반영하면서도 모수(상관모수)가 <img src="https://latex.codecogs.com/png.latex?%5Crho"> 하나라 단순하고, 보고/해석이 깔끔하며 효율이 좋아질 수 있다.</li>
<li>단점: 시간 간격에 따라 상관이 달라지는 패턴(예: 가까운 시점끼리 더 강한 상관)을 표현하지 못한다.</li>
</ul>
</section>
<section id="상관구조의-선택" class="level2">
<h2 class="anchored" data-anchor-id="상관구조의-선택">상관구조의 선택</h2>
<p>작업 상관구조를 하나로 단정하기 어렵다면, 가장 자연스러운 제시는 <strong>(1) 주 분석의 기본 구조를 정하고, (2) 다른 구조로 민감도 분석을 수행해 결론의 견고함을 확인</strong>하는 방식이다. 여기서 중요한 것은 “exchangeable이 이론적으로 더 좋아 보인다”가 아니라</p>
<p><strong>데이터가 exchangeable을 추정할 만큼의 정보가 있고 적합이 안정적인지</strong>다. 다음과 같은 상황에서는 exchangeable을 쓰는 것이 현실적으로 어렵거나(수렴/추정 불안정), <img src="https://latex.codecogs.com/png.latex?%5Crho"> 추정 자체가 의미 있게 되기 힘들다.</p>
<ul>
<li>한 환자당 반복측정 횟수가 매우 적어(<img src="https://latex.codecogs.com/png.latex?n_i">가 작음) 환자 내 상관을 추정할 정보가 거의 없는 경우</li>
<li>클러스터 수(환자 수)가 작거나, 불균형/결측이 심해 <img src="https://latex.codecogs.com/png.latex?%5Crho"> 추정이 불안정해지는 경우</li>
<li>적합 과정에서 경고/수렴 문제가 반복되거나, 추정된 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Crho%7D">가 극단값(예: <img src="https://latex.codecogs.com/png.latex?%5Cpm%201">에 근접)으로 튀는 경우</li>
<li>robust SE 대비 model-based SE가 비정상적으로 괴리되는 등, 상관모수 추정이 불안정하다고 판단되는 경우</li>
</ul>
<p>이런 경우에는 independence를 “대안”이 아니라 기본 상관구조로 두는 것이 더 타당하다. GEE의 목적이 평균효과 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">의 추정과 강건한 추론에 있는 만큼, 불안정한 <img src="https://latex.codecogs.com/png.latex?%5Crho">를 억지로 추정하는 것보다 가정을 최소화하고 robust SE로 추론하는 전략이 리뷰어 관점에서도 납득 가능하다. 정리하면 선택 흐름은 다음 두 가지 중 하나로 가져가면 된다.</p>
<p><strong>- exchangeable이 안정적으로 적합되고 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Crho%7D"> 추정이 합리적인 경우 경우: </strong> exchangeable 상관구조를 사용한다.</p>
<p><strong>- exchangeable이 불안정하거나(수렴/경고/극단적 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Crho%7D">) 상관 추정 정보가 부족한 경우: </strong> 주 분석을 independence로 두고, 가능하다면 exchangeable 시도 결과를 민감도 분석(또는 부록)으로만 언급한다.</p>
<p>결론적으로, Outcome의 비정규성이 우려되는 상황이라면 정규 우도(Likelihood)에 기반한 LMM의 제약에서 벗어나, Quasi-likelihood와 Robust 표준오차를 제공하는 GEE를 적극적으로 고려한다. 이때 상관구조의 선택은 단순히 ’구조의 정답’을 찾는 과정이라기보다, 데이터의 복잡도와 수렴 안정성을 고려하는 판단의 과정이다. 모델이 안정적이라면 Exchangeable을, 추정 정보가 부족하거나 수렴이 불안정하다면 Independence를 기본으로 두되, 상관구조 변경에도 결론이 일관되게 유지되는지 확인하는 과정을 거친다면 더 신뢰도 높은 결과를 도출할 수 있을 것이다.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{oh2026,
  author = {Oh, Sangho},
  title = {비정규 {분포의} {데이터에} {GEE를} {사용할} {수} {있을까?}},
  date = {2026-01-02},
  url = {https://blog.zarathu.com/posts/2026-01-02-LMM, GEE/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-oh2026" class="csl-entry quarto-appendix-citeas">
Oh, Sangho. 2026. <span>“비정규 분포의 데이터에 Gee를 사용할 수
있을까?”</span> January 2, 2026. <a href="https://blog.zarathu.com/posts/2026-01-02-LMM, GEE/">https://blog.zarathu.com/posts/2026-01-02-LMM,
GEE/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <guid>https://blog.zarathu.com/posts/2026-01-02-LMM, GEE/</guid>
  <pubDate>Fri, 02 Jan 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>마우스 드래그로 끝내는 PDF 테이블 추출: tabulapdf</title>
  <dc:creator>Wonbin Hahn</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-12-17-tabulapdf/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="들어가며" class="level2"><h2 class="anchored" data-anchor-id="들어가며">들어가며</h2>
<p>보통 데이터는 csv, xlsx, 혹은 sas나 rtf 형식으로 받습니다. 이런 형식들은 읽어오기도 편하고, 데이터의 무결성도 유지하는 확실한 방법들입니다.</p>
<p>하지만 간혹 pdf 파일에 있는 데이터를 가져와야 하는 경우가 있습니다. 이럴 때는 어떻게 해야 할까요?</p>
<p>저 역시 그런 상황을 겪었는데, 복사-붙여넣기도 제대로 되지 않아 많이 당황했습니다. 수많은 데이터가 담긴 pdf를 일일이 타이핑하는 것은 부정확할 뿐만 아니라 시간도 너무 오래 걸리는 일입니다.</p>
<p>바로 이때 나타난 해결사, <strong>pdf에 있는 테이블을 추출해주는</strong> <code>tabulapdf</code> 패키지를 소개합니다.</p>
</section><section id="패키지-설치-및-로드" class="level2"><h2 class="anchored" data-anchor-id="패키지-설치-및-로드">패키지 설치 및 로드</h2>
<p><code>tabulapdf</code>는 Java를 기반으로 만들어진 패키지입니다. 다행히 별도의 복잡한 Java 설정 없이 패키지 설치만으로 대부분 작동합니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install.packages("tabulapdf")</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://docs.ropensci.org/tabulapdf/">tabulapdf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://r-datatable.com">data.table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
</section><section id="기본-사용법" class="level2"><h2 class="anchored" data-anchor-id="기본-사용법">기본 사용법</h2>
<p>가장 기본적이면서 강력한 함수는 <code>extract_tables</code>입니다. 이 함수는 pdf 파일의 모든 페이지를 스캔하며 테이블을 인식하고, 이를 R에서 다루기 쉬운 tibble 등의 리스트 형태로 반환해 줍니다. 한 페이지에 여러 테이블이 있더라도 일일이 다 스캔하여 반환해줍니다.</p>
<p>다음은 패키지에 내장된 예제 <code>mtcars.pdf</code>를 활용한 코드입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 예시 데이터</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/system.file.html">system.file</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"examples"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"mtcars.pdf"</span>, package <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tabulapdf"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 전체 페이지 추출</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>[[1]]
# A tibble: 5 × 12
  model          mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
  &lt;chr&gt;        &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
1 Mazda RX4     21       6   160   110  3.9   2.62  16.5     0     1     4     4
2 Mazda RX4 W…  21       6   160   110  3.9   2.88  17.0     0     1     4     4
3 Datsun 710    22.8     4   108    93  3.85  2.32  18.6     1     1     4     1
4 Hornet 4 Dr…  21.4     6   258   110  3.08  3.21  19.4     1     0     3     1
5 Hornet Spor…  18.7     8   360   175  3.15  3.44  17.0     0     0     3     2

[[2]]
# A tibble: 1 × 5
  Sepal.Length                   Sepal.Width    Petal.Length Petal.Width Species
  &lt;chr&gt;                          &lt;chr&gt;          &lt;chr&gt;        &lt;chr&gt;       &lt;chr&gt;  
1 "5.10\r4.90\r4.70\r4.60\r5.00" "3.50\r3.00\r… "1.40\r1.40… "0.20\r0.2… "setos…

[[3]]
# A tibble: 5 × 3
    len supp   dose
  &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt;
1   4.2 VC      0.5
2  11.5 VC      0.5
3   7.3 VC      0.5
4   5.8 VC      0.5
5   6.4 VC      0.5</code></pre>
</div>
</div>
<p>이런 기본 기능만으로도 훌륭하지만 모든 페이지를 탐색할 필요 없이 특정 페이지만 보고 싶다면 <strong>pages</strong> 옵션을 사용하면 됩니다. 처리 시간도 단축되고, 추출된 결과물을 확인하기도 훨씬 수월해집니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 특정 페이지 추출</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span>, pages <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>[[1]]
# A tibble: 5 × 12
  model          mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
  &lt;chr&gt;        &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
1 Mazda RX4     21       6   160   110  3.9   2.62  16.5     0     1     4     4
2 Mazda RX4 W…  21       6   160   110  3.9   2.88  17.0     0     1     4     4
3 Datsun 710    22.8     4   108    93  3.85  2.32  18.6     1     1     4     1
4 Hornet 4 Dr…  21.4     6   258   110  3.08  3.21  19.4     1     0     3     1
5 Hornet Spor…  18.7     8   360   175  3.15  3.44  17.0     0     0     3     2</code></pre>
</div>
</div>
<p>여러 페이지를 원한다면 벡터로 입력하면 됩니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 여러 페이지 추출</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span>, pages <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>[[1]]
# A tibble: 5 × 12
  model          mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
  &lt;chr&gt;        &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
1 Mazda RX4     21       6   160   110  3.9   2.62  16.5     0     1     4     4
2 Mazda RX4 W…  21       6   160   110  3.9   2.88  17.0     0     1     4     4
3 Datsun 710    22.8     4   108    93  3.85  2.32  18.6     1     1     4     1
4 Hornet 4 Dr…  21.4     6   258   110  3.08  3.21  19.4     1     0     3     1
5 Hornet Spor…  18.7     8   360   175  3.15  3.44  17.0     0     0     3     2

[[2]]
# A tibble: 5 × 3
    len supp   dose
  &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt;
1   4.2 VC      0.5
2  11.5 VC      0.5
3   7.3 VC      0.5
4   5.8 VC      0.5
5   6.4 VC      0.5</code></pre>
</div>
</div>
</section><section id="테이블-형태에-따른-설정" class="level2"><h2 class="anchored" data-anchor-id="테이블-형태에-따른-설정">테이블 형태에 따른 설정</h2>
<p><code>extract_tables</code> 를 이용하여 테이블을 가져올 때 크게 두 가지 방식이 있습니다.</p>
<ul>
<li>
<strong><code>lattice</code></strong> (기본값): 테이블에 테두리나 선이 명확하게 있는 경우 사용합니다.</li>
<li>
<strong><code>stream</code></strong>: 테이블에 테두리나 선이 없는 경우 사용합니다.</li>
</ul>
<p>일반적인 테이블은 <code>method = "lattice"</code> 로 인식이 잘 됩니다. 여기서 일반적인 테이블이란 테두리와 경계선이 모두 그려진 규칙적인 테이블을 의미합니다.</p>
<p>하지만 분명히 행렬이거나 테이블은 맞는데, 선이 없는 디자인의 테이블이라면 알고리즘이 표를 인식하기 힘들어합니다. 그럴 때 <code>method = "stream"</code>을 씁니다. 이 방법은 문자열 사이의 공백을 기준으로 컬럼을 구분하기 때문에, 선이 없거나 한 페이지에 열 개수가 다른 여러 테이블이 섞여 있을 때 훨씬 정확한 결과를 얻을 수 있습니다.</p>
<p>밑에 예시를 보시면 기본 메소드로는 테이블을 못 읽고, <code>method = "stream"</code> 인 경우에 제대로 인식합니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 선이 없는 테이블 추출 시 (lattice) 사용 - 부정확</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span>, pages <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, method <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lattice"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 1 × 5
  Sepal.Length                   Sepal.Width    Petal.Length Petal.Width Species
  &lt;chr&gt;                          &lt;chr&gt;          &lt;chr&gt;        &lt;chr&gt;       &lt;chr&gt;  
1 "5.10\r4.90\r4.70\r4.60\r5.00" "3.50\r3.00\r… "1.40\r1.40… "0.20\r0.2… "setos…</code></pre>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 선이 없는 테이블 추출 시 (stream) 사용 - 정확</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span>, pages <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, method <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"stream"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 5 × 5
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
         &lt;dbl&gt;       &lt;dbl&gt;        &lt;dbl&gt;       &lt;dbl&gt; &lt;chr&gt;  
1          5.1         3.5          1.4         0.2 setosa 
2          4.9         3            1.4         0.2 setosa 
3          4.7         3.2          1.3         0.2 setosa 
4          4.6         3.1          1.5         0.2 setosa 
5          5           3.6          1.4         0.2 setosa </code></pre>
</div>
</div>
</section><section id="마우스로-드래그해서-가져오기" class="level2"><h2 class="anchored" data-anchor-id="마우스로-드래그해서-가져오기">마우스로 드래그해서 가져오기</h2>
<p>만약 페이지 전체가 아니라 <strong>특정 구역의 테이블만</strong> 콕 집어서 가져오고 싶다면 어떻게 해야 할까요?</p>
<p>이 패키지의 가장 혁신적인 기능인 <code>locate_areas</code>와 <code>extract_areas</code>를 사용하면 됩니다.</p>
<p>이 함수들을 실행하면 RStudio의 Viewer 패널에 pdf가 한 페이지씩 렌더링됩니다. 실제로 해보시면 느끼시겠지만, Shiny 인터페이스와 연동되어 작동하는 방식이 굉장히 직관적입니다.</p>
<p>사용자는 <strong>마우스로 원하는 구역을 드래그하고 오른쪽 위의 ’완료’를 누르기</strong>만 하면 됩니다. 함수에 여러 페이지를 넣었다면, 모든 페이지에서 드래그 후 완료를 눌러야 테이블 추출 작업이 끝나고 반환해줍니다. 드래그할 내용이 없는 페이지는 바로 완료 누르고 넘어가면 됩니다.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-12-17-tabulapdf/img/viewer.png" class="img-fluid figure-img"></p>
<figcaption>PDF 테이블 추출 예시</figcaption></figure>
</div>
<p>마우스로 드래그해서 가져오는 두 함수의 기능은 각각 다음과 같습니다.</p>
<ul>
<li><p><code>locate_areas</code>: 드래그한 구역의 <strong>좌표</strong>를 반환합니다.</p></li>
<li><p><code>extract_areas</code>: 드래그한 구역의 <strong>데이터(테이블)</strong>를 즉시 반환합니다.</p></li>
</ul>
<section id="좌표-확인하기" class="level3"><h3 class="anchored" data-anchor-id="좌표-확인하기">좌표 확인하기</h3>
<p><code>locate_areas</code>는 구역을 고르면 그 구역의 <strong>각 꼭짓점의 좌표</strong>를 반환해줍니다. 한 페이지에 하나의 구역 설정을 할 수 있습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 실행 후 영역 드래그 -&gt; 좌표 반환</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">locate_areas</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-12-17-tabulapdf/img/locate_areas.png" class="img-fluid figure-img"></p>
<figcaption>locate_areas 결과 예시</figcaption></figure>
</div>
</section><section id="테이블-바로-추출하기" class="level3"><h3 class="anchored" data-anchor-id="테이블-바로-추출하기">테이블 바로 추출하기</h3>
<p>좌표 없이 바로 드래그를 하고 싶으시다면 <code>extract_areas</code>로 구역을 정하면 바로 그 테이블을 반환합니다. 마찬가지로 한 페이지에 하나의 구역 설정을 할 수 있습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 실행 후 영역 드래그 -&gt; 테이블 반환</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_areas</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
</section></section><section id="복잡한-헤더-구조-해결하기" class="level2"><h2 class="anchored" data-anchor-id="복잡한-헤더-구조-해결하기">복잡한 헤더 구조 해결하기</h2>
<p>여러 종류의 pdf를 다루다 보면 테이블 구조가 복잡해서 인식이 잘 안될 때가 있습니다. 특히 <strong>데이터나 헤더가 여러 열에 걸쳐 병합되어 있는 경우</strong>, <code>extract_tables</code>나 <code>extract_areas</code>를 쓰더라도 열이 뭉개지거나 하나로 합쳐져서 반환되는 문제가 발생합니다.</p>
<p>이럴 때는 문제가 되는 <strong>헤더 부분을 제외하고, 규칙적인 데이터 영역만 스캔</strong>하는 것이 좋습니다. 이렇게 데이터만 먼저 긁어오고, 헤더는 R에서 후작업으로 추가하는 방식이 더 정확합니다.</p>
<p>다음은 패키지에 내장된 예제 <code>covid.pdf</code>를 활용한 코드입니다. 제일 상단 행이 병합되어 있어 그대로 긁어오면 열이 깨집니다. 따라서 데이터 영역만 좌표로 지정하고 <code>col_names = FALSE</code> 옵션을 주어 깔끔하게 가져오는 방법을 씁니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 예시 데이터</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/system.file.html">system.file</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"examples"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"covid.pdf"</span>, package <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tabulapdf"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># locate_areas(f)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 헤더를 제외한 데이터 영역만 추출 (col_names = FALSE)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covid_list</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">extract_tables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">f</span>, pages <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, </span>
<span>    guess <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># area를 좌표로 입력할 때 꼭 쓰기</span></span>
<span>    col_names <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 헤더 자동 인식 끄기</span></span>
<span>    area <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">120.6259</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">92.2500</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">373.5551</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">633.3750</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 좌표</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covid_dt</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covid_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 필요한 경우 여기서 setnames() 등을 통해 컬럼명 지정</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># setnames(covid_dt, c("Col1", "Col2", ...)) </span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covid_dt</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>                       X1         X2     X3         X4           X5
                   &lt;char&gt;     &lt;char&gt; &lt;char&gt;     &lt;char&gt;       &lt;char&gt;
 1:               Regione Remdesivir   Inc% Remdesivir Molnupiravir
 2:               Abruzzo      2.343   3,0%         27           23
 3:            Basilicata        927   1,2%          -            -
 4:              Calabria      1.797   2,3%          -            -
 5:              Campania      3.289   4,1%         20           12
 6:        Emilia Romagna      7.945  10,0%        244           17
 7: Friuli Venezia Giulia      1.063   1,3%         11           59
 8:                 Lazio     11.206  14,1%         63          219
 9:               Liguria      5.332   6,7%         87          207
10:             Lombardia     12.089  15,3%        239          114
11:                Marche      3.739   4,7%         10           85
12:                Molise         42   0,1%          -            5
13:              Piemonte      5.593   7,1%         50          203
14:  Prov. Auton. Bolzano        188   0,2%          -           11
15:   Prov. Auton. Trento        211   0,3%          -            3
16:                Puglia      4.439   5,6%         13           66
17:              Sardegna        757   1,0%          -           29
18:               Sicilia      4.383   5,5%          2           30
19:               Toscana      5.916   7,5%        106          161
20:                Umbria      1.583   2,0%          9           42
21:         Valle D'aosta        351   0,4%          -           20
22:                Veneto      6.072   7,7%         20          186
23:                Italia     79.265 100,0%        901        1.492
                       X1         X2     X3         X4           X5
                    X6           X7             X8
                &lt;char&gt;       &lt;char&gt;         &lt;char&gt;
 1: Totale per regione Remdesivir % Molnupiravir %
 2:                 50         3,0%           1,5%
 3:                  -         0,0%           0,0%
 4:                  -         0,0%           0,0%
 5:                 32         2,2%           0,8%
 6:                261        27,1%           1,1%
 7:                 70         1,2%           4,0%
 8:                282         7,0%          14,7%
 9:                294         9,7%          13,9%
10:                353        26,5%           7,6%
11:                 95         1,1%           5,7%
12:                  5         0,0%           0,3%
13:                253         5,5%          13,6%
14:                 11         0,0%           0,7%
15:                  3         0,0%           0,2%
16:                 79         1,4%           4,4%
17:                 29         0,0%           1,9%
18:                 32         0,2%           2,0%
19:                267        11,8%          10,8%
20:                 51         1,0%           2,8%
21:                 20         0,0%           1,3%
22:                206         2,2%          12,5%
23:              2.393        37,7%          62,3%
                    X6           X7             X8</code></pre>
</div>
</div>
</section><section id="일관성-있는-테이블-추출을-위한-팁" class="level2"><h2 class="anchored" data-anchor-id="일관성-있는-테이블-추출을-위한-팁">일관성 있는 테이블 추출을 위한 팁</h2>
<p>두 가지 접근 방식이 있습니다.</p>
<ul>
<li><p><strong>일회성 작업</strong>: 단순히 엑셀이나 CSV로 한 번 다운로드하는 것이 목적이라면 <code>extract_areas</code>로 바로 데이터를 뽑는 것이 편합니다.</p></li>
<li><p><strong>반복 작업</strong>: 만약 코드를 저장해두고 계속 반복해서 실행해야 한다면, <code>locate_areas</code>를 한 번 사용해 좌표를 먼저 얻는 것을 추천합니다. 얻어낸 좌표를 <code>extract_tables(..., area = ...)</code> 에 입력하여 사용하면, <strong>매번 마우스로 드래그할 필요 없이 일관성 있는 결과</strong>를 얻을 수 있습니다.</p></li>
</ul></section><section id="마무리" class="level2"><h2 class="anchored" data-anchor-id="마무리">마무리</h2>
<p><code>tabulapdf</code> 패키지 덕분에 PDF에서 데이터를 추출하는 과정이 획기적으로 간단해졌습니다. 단순히 좌표를 입력하는 것을 넘어 GUI 환경에서 직접 드래그하여 구역을 설정할 수 있다는 점이 매우 편리했습니다. 테이블이 복잡할수록 읽는 데에 어려움이 있지만, 규칙적으로 읽어주기 때문에 가져올 수 있는 부분이라도 가져온 후에 전처리 하는 것이 중요하다고 생각합니다. 앞으로 다른 패키지들도 이러한 기능을 도입하여 분석가들의 작업 효율을 높여주길 기대해 봅니다.</p>
</section><section id="참고-자료" class="level2"><h2 class="anchored" data-anchor-id="참고-자료">참고 자료</h2>
<p>이 글에서 소개한 <code>tabulapdf</code> 패키지에 대한 더 자세한 정보나 소스 코드는 아래 링크에서 확인하실 수 있습니다.</p>
<ul>
<li>
<a href="https://github.com/ropensci/tabulapdf">tabulapdf GitHub Repository</a>: 패키지 개발자의 깃허브입니다.</li>
<li>
<a href="https://cran.r-project.org/web/packages/tabulapdf/index.html">CRAN Package Page</a>: CRAN 공식 페이지입니다.</li>
</ul>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hahn2025,
  author = {Hahn, Wonbin},
  title = {마우스 {드래그로} {끝내는} {PDF} {테이블} {추출:} Tabulapdf},
  date = {2025-12-17},
  url = {https://blog.zarathu.com/posts/2025-12-17-tabulapdf/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hahn2025" class="csl-entry quarto-appendix-citeas">
Hahn, Wonbin. 2025. <span>“마우스 드래그로 끝내는 PDF 테이블 추출:
Tabulapdf.”</span> December 17, 2025. <a href="https://blog.zarathu.com/posts/2025-12-17-tabulapdf/">https://blog.zarathu.com/posts/2025-12-17-tabulapdf/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <category>Rpackage</category>
  <guid>https://blog.zarathu.com/posts/2025-12-17-tabulapdf/</guid>
  <pubDate>Wed, 17 Dec 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-12-17-tabulapdf/img/logo.png" medium="image" type="image/png" height="145" width="144"/>
</item>
<item>
  <title>코호트 연구의 핵심, 대조군의 Index date 설정</title>
  <dc:creator>Wonbin Hahn</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-11-18-control_index/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="introduction" class="level1"><h1>Introduction</h1>
<section id="rct-rwd-그리고-관찰-연구" class="level2"><h2 class="anchored" data-anchor-id="rct-rwd-그리고-관찰-연구">RCT, RWD, 그리고 관찰 연구</h2>
<p>의학 연구에서 특정 치료법이나 요인의 인과관계를 규명하는 가장 강력한 표준 방법은 <strong>무작위 대조 시험(Randomized Controlled Trial, RCT)</strong>입니다. RCT는 연구 대상을 무작위로 치료군과 대조군에 배정하여 두 집단 간의 편향을 최소화합니다.</p>
<p>하지만 RCT 가 항상 수행 가능한 것은 아닙니다. 비용과 시간의 문제도 있고, 윤리적인 문제도 베재할 수 없습니다. 예를 들어 담배와 같은 유해한 요인을 강제로 노출시킬 수 없습니다. 또한 엄격하게 통제된 환경에서 진행되기 때문에 현실의 다양한 환자들에게 동일하게 적용되기 어렵다는는 한계도 있습니다.</p>
<p>이런 경우에 <strong>실제 임상 데이터(Real-World Data, RWD)</strong>가 쓰입니다. RWD는 건강보험심사평가원 데이터, 국민건강보험공단 데이터 등 실제 진료 환경에서 수집된 방대한 데이터를 의미합니다.</p>
<p>이러한 RWD를 활용하는 대표적인 연구 방법이 바로 <strong>관찰 연구(Observational Study)</strong>이며, <strong>코호트 연구(Cohort Study)</strong>는 그 핵심 방법론 중 하나입니다.</p>
</section><section id="cohort-study-란" class="level2"><h2 class="anchored" data-anchor-id="cohort-study-란">Cohort Study 란?</h2>
<p><strong>코호트 연구(Cohort Study)</strong>는 특정 요인에 노출된 집단(exposed group)과 노출되지 않은 집단(unexposed group)을 시간의 흐름에 따라 추적하며 특정 질병이나 결과(outcome)의 발생률을 비교하는 관찰 연구 방법입니다.</p>
<p>잘 설계된 코호트 연구는 강력한 결과를 제공할 수 있습니다. 코호트 연구에서는 먼저 특정 노출이나 사건을 기준으로 질병이나 결과가 없는 연구 집단을 식별합니다. 그리고 이들을 시간의 흐름에 따라 추적하며 관심 있는 질병이나 결과가 발생할 때까지 관찰합니다. 이처럼 <strong>노출이 결과보다 먼저 식별되기 때문에</strong>, 코호트 연구는 인과성을 평가할 수 있는 시간적 기준을 가지며, 관찰 연구 중에서는 강력한 과학적 증거를 제공합니다.</p>
</section><section id="cohort-vs.-case-control" class="level2"><h2 class="anchored" data-anchor-id="cohort-vs.-case-control">Cohort vs.&nbsp;Case-Control</h2>
<p>관찰 연구에는 코호트 연구 외에도 환자-대조군 연구(Case-Control Study)가 있습니다. 두 연구는 RWD를 사용한다는 공통점이 있지만, 연구의 시작점과 방향이 정반대입니다.</p>
<div class="cell">
<div class="cell-output-display">
<div class="tabwid">
<style>.cl-9d48335e{table-layout:auto;width:100%;}.cl-9d45b0e8{font-family:'Helvetica';font-size:11pt;font-weight:bold;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-9d45b0f2{font-family:'Helvetica';font-size:11pt;font-weight:normal;font-style:normal;text-decoration:none;color:rgba(0, 0, 0, 1.00);background-color:transparent;}.cl-9d46a912{margin:0;text-align:center;border-bottom: 0 solid rgba(0, 0, 0, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0 solid rgba(0, 0, 0, 1.00);border-right: 0 solid rgba(0, 0, 0, 1.00);padding-bottom:5pt;padding-top:5pt;padding-left:5pt;padding-right:5pt;line-height: 1;background-color:transparent;}.cl-9d46b2ea{background-color:rgba(242, 242, 242, 1.00);vertical-align: middle;border-bottom: 0.75pt solid rgba(102, 102, 102, 1.00);border-top: 0.75pt solid rgba(102, 102, 102, 1.00);border-left: 0.75pt solid rgba(102, 102, 102, 1.00);border-right: 0.75pt solid rgba(102, 102, 102, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-9d46b2eb{background-color:transparent;vertical-align: middle;border-bottom: 0.75pt solid rgba(102, 102, 102, 1.00);border-top: 0 solid rgba(0, 0, 0, 1.00);border-left: 0.75pt solid rgba(102, 102, 102, 1.00);border-right: 0.75pt solid rgba(102, 102, 102, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}.cl-9d46b2ec{background-color:transparent;vertical-align: middle;border-bottom: 0.75pt solid rgba(102, 102, 102, 1.00);border-top: 0.75pt solid rgba(102, 102, 102, 1.00);border-left: 0.75pt solid rgba(102, 102, 102, 1.00);border-right: 0.75pt solid rgba(102, 102, 102, 1.00);margin-bottom:0;margin-top:0;margin-left:0;margin-right:0;}</style>
<table data-quarto-disable-processing="true" class="cl-9d48335e">
<thead><tr style="overflow-wrap:break-word;">
<th class="cl-9d46b2ea"><p class="cl-9d46a912"><span class="cl-9d45b0e8">비교 항목</span></p></th>
<th class="cl-9d46b2ea"><p class="cl-9d46a912"><span class="cl-9d45b0e8">Cohort Study</span></p></th>
<th class="cl-9d46b2ea"><p class="cl-9d46a912"><span class="cl-9d45b0e8">Case-Control Study</span></p></th>
</tr></thead>
<tbody>
<tr style="overflow-wrap:break-word;">
<td class="cl-9d46b2eb"><p class="cl-9d46a912"><span class="cl-9d45b0e8">연구 시작점</span></p></td>
<td class="cl-9d46b2eb"><p class="cl-9d46a912"><span class="cl-9d45b0f2">노출 (Exposure)</span></p></td>
<td class="cl-9d46b2eb"><p class="cl-9d46a912"><span class="cl-9d45b0f2">결과 (Outcome)</span></p></td>
</tr>
<tr style="overflow-wrap:break-word;">
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0e8">연구 방향</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">미래 (Forward-looking)</span><br><span class="cl-9d45b0f2">(노출 → 결과)</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">과거 (Backward-looking)</span><br><span class="cl-9d45b0f2">(결과 → 노출)</span></p></td>
</tr>
<tr style="overflow-wrap:break-word;">
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0e8">연구 집단</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">A약물 복용군 (노출군)</span><br><span class="cl-9d45b0f2">vs.</span><br><span class="cl-9d45b0f2">A약물 미복용군 (대조군)</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">B질병 환자군 (Case)</span><br><span class="cl-9d45b0f2">vs.</span><br><span class="cl-9d45b0f2">B질병 없는 대조군 (Control)</span></p></td>
</tr>
<tr style="overflow-wrap:break-word;">
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0e8">주요 질문</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">"A약 복용 시 B질병 발생</span><br><span class="cl-9d45b0f2">위험이 얼마나 변하는가?"</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">"B질병 환자들이 과거에</span><br><span class="cl-9d45b0f2">A약을 얼마나 더 많이 복용했는가?"</span></p></td>
</tr>
<tr style="overflow-wrap:break-word;">
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0e8">주요 측정 지표</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">위험비 (RR, HR)</span></p></td>
<td class="cl-9d46b2ec"><p class="cl-9d46a912"><span class="cl-9d45b0f2">교차비 (OR)</span></p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</section><section id="cohort-study-는-어떻게-진행되는가" class="level2"><h2 class="anchored" data-anchor-id="cohort-study-는-어떻게-진행되는가">Cohort Study 는 어떻게 진행되는가?</h2>
<p>일반적으로 코호트 연구는 다음과 같은 단계로 진행됩니다.</p>
<ol type="1">
<li>
<strong>연구 질문 설정</strong>: 명확한 가설을 설정합니다. (예: “A 약물 복용이 B 질병 발생 위험을 낮추는가?”)</li>
<li>
<strong>연구 집단 선정</strong>:
<ul>
<li>
<strong>치료군/노출군 (Treatment/Exposed Group)</strong>: A 약물을 복용한 사람</li>
<li>
<strong>대조군/비노출군 (Control/Unexposed Group)</strong>: A 약물을 복용하지 않은 사람</li>
</ul>
</li>
<li>
<strong>Index Date 정의</strong>: 연구의 ’시작일’을 정의합니다. 이는 <strong>결과가 발생하기 전</strong>이어야 하며, 두 집단 간의 추적 기간을 동일하게 설정하는 기준이 됩니다.</li>
<li>
<strong>추적 관찰 (Follow-up)</strong>: 두 집단을 일정 기간 추적하며 B 질병 발생 여부를 관찰합니다.</li>
<li>
<strong>결과 분석</strong>: 두 집단의 B 질병 발생률을 비교하여 위험비(Hazard Ratio, Relative Risk 등)를 계산합니다.</li>
</ol>
<p>이 글에서는 3번 단계의 Index Date를 정의하는 문제, 특히 <strong>대조군의 Index Date를 어떻게 설정</strong>해야 하는지에 초점을 맞춥니다.</p>
</section><section id="propensity-score-matching-vs.-exact-matching" class="level2"><h2 class="anchored" data-anchor-id="propensity-score-matching-vs.-exact-matching">Propensity Score Matching vs.&nbsp;Exact matching</h2>
<p>코호트 연구에서 치료군과 대조군을 비교할 때 흔히 잘못 생각할 수 있는 부분 중 하나는 대조군이 치료군과 <strong>모든 점에서 달라야 한다고</strong> 생각하는 것입니다. (예: 치료군은 환자, 대조군은 ‘완벽하게 건강한 사람’)</p>
<p>오히려 그 반대입니다. 공정한 비교를 위해서는 <strong>대조군을 치료군과 최대한 비슷하게 설정</strong>하는 것이 중요합니다.</p>
<p>왜 그럴까요?</p>
<p>대조군 역시 ‘만약 상황이 달랐다면’ 치료군처럼 <strong>노출될 가능성이 잠재적으로 있었던 집단</strong>이어야 하기 때문입니다. 예를 들어 특정 항암제의 효과를 본다면, 대조군 역시 해당 암을 진단받았으나 그 약만 투여받지 않은 환자여야지, 암 자체가 없는 건강한 사람이면 비교가 어렵습니다.</p>
<p>이렇게 두 집단의 특성(나이, 성별, 기저질환 중증도 등)이 비슷해야, 우리가 관찰한 결과의 차이가 그 노출(치료) 요인 하나에서 비롯되었다고 주장할 수 있습니다. 비교할 유사 집단을 찾는 과정을 <strong>매칭(Matching)</strong>이라고 합니다.</p>
<ul>
<li>
<p><strong>Exact Matching</strong>:</p>
<p>나이, 성별 등 <strong>모든 매칭 변수의 값이 정확히 일치</strong>하는 대상을 1:1 또는 1:N으로 짝짓는 방법입니다. 직관적이지만 변수가 많아지면 매칭 대상을 찾기 어려워집니다.</p>
</li>
<li>
<p><strong>Propensity Score Matching (PSM)</strong>:</p>
<p>나이, 성별, 기저질환 등 수많은 변수를 사용하여 “<strong>그 사람이 치료(Treatment)를 받을 확률</strong>”을 하나의 점수(Propensity Score)로 계산합니다. 그 후, <strong>점수가 비슷한</strong> 치료군과 대조군을 매칭합니다. 많은 변수를 하나의 차원으로 줄여 매칭 효율성을 높이는 강력한 방법입니다.</p>
</li>
</ul></section><section id="treatment-vs.-control-group-index-date-설정" class="level2"><h2 class="anchored" data-anchor-id="treatment-vs.-control-group-index-date-설정">Treatment vs.&nbsp;Control Group: Index Date 설정</h2>
<p><strong>치료군(Treatment Group)</strong>은 보통 Index Date가 명확합니다. 질병을 처음 진단받은 날, 또는 특정 약을 처음 투여받은 날짜로 설정하면 됩니다.</p>
<p><strong>하지만 대조군(Control Group)은 어떨까요?</strong></p>
<p>만약 대조군이 ‘다른 질병’ 진단군이거나 ‘다른 약’ 투여군이라면 그 진단/투여 날짜를 Index Date로 사용하면 됩니다. 하지만 대조군이 해당 질병이 없거나 해당 약을 투여하지 않은 건강한 사람이라면 명확한 시작일이 없습니다. 이때 대조군의 Index Date를 어떻게 정해야 할까요? 이는 연구의 타당성에 매우 중요한 영향을 미칩니다.</p>
<p>여러 논문을 리뷰해본 결과 크게 3가지 방법이 주로 사용됩니다.</p>
</section><section id="대조군-index-date-설정-방법-3가지-전략" class="level2"><h2 class="anchored" data-anchor-id="대조군-index-date-설정-방법-3가지-전략">대조군 Index Date 설정 방법: 3가지 전략</h2>
<section id="방법-1-치료군의-index-date를-그대로-사용" class="level3"><h3 class="anchored" data-anchor-id="방법-1-치료군의-index-date를-그대로-사용">방법 1: 치료군의 Index Date를 그대로 사용</h3>
<p>가장 일반적이고 직관적인 방법입니다. 나이, 성별, 지역 등 공변량(covariate)으로 치료군과 대조군을 매칭한 후, <strong>매칭된 치료군의 Index Date(진단/투여일)를 대조군의 Index Date로 그대로 할당</strong>하는 방식입니다.</p>
<ul>
<li>
<strong>장점</strong>:
<ul>
<li>개념이 단순하고 구현이 쉽습니다.</li>
<li>치료군과 대조군이 추적 관찰을 시작하는 달력상의 시점(calendar time)이 동일해져, 계절적 요인이나 시간에 따른 의료 시스템 변화 등(time-related biases)을 통제하기 용이합니다.</li>
</ul>
</li>
<li>
<strong>단점</strong>:
<ul>
<li>매칭 시점에 대조군이 살아있고, 관찰 가능(under follow-up)해야 하며, 해당 질병이 없어야 한다는 조건을 만족해야 합니다.</li>
<li>1:N 매칭 시, 한 명의 치료군 날짜를 N명의 대조군이 공유하게 됩니다.</li>
</ul>
</li>
</ul>
<blockquote class="blockquote">
<p><strong>[예시 논문 1] Forbes, Harriet et al.&nbsp;(eClinicalMedicine, 2023)</strong><sup>1</sup></p>
<p>Matching was done <strong>without replacement</strong> and cancer survivors diagnosed <strong>earliest in calendar time were matched first</strong> (greedy matching approach), to avoid time-related biases. For each cancer survivor, we <strong>randomly selected up to 10 cancer-free comparators from the overall unexposed population</strong>, matched at index date on birth year ( ± 3 years, with closer matches given preference), sex and GP practice; comparators had to be under follow-up and have no cancer history (besides NMSC) on their index date (i.e., on the diagnosis date of the matched cancer case). The matching ratio of 1:10 was chosen as a compromise between maximising precision and minimising exclusions of some patients due to running out of available matches. Similar to cancer survivors, cancer-free comparators were required to have ≥1-year continuous registration prior to index date.</p>
</blockquote>
<p>암 환자 목록을 암 진단 날짜순으로 나열한 후 그 순서대로 대조군에서 10명씩 매칭한 것입니다. 매칭할 때는 그 해당 날짜에 유효한 정보를 갖고 있는 대조군 모두가 대상이 되며, 한 번 매칭된 대조군은 다시 복원되지 않는 greedy matching approach를 씁니다. Index date는 매칭된 암환자의 암 진단 날짜를 그대로 갖고옵니다. 여기서는 1:10 비율로 매칭을 한 것을 볼 수 있습니다.</p>
<blockquote class="blockquote">
<p><strong>[예시 논문 2] Tuberculosis and Risk of Ischemic Stroke (Stroke, 2017)</strong><sup>2</sup></p>
<p>We obtained an initial pool of ≈1 million non-tuberculosis cases for matching. Among them, we serially matched all tuberculosis cases diagnosed in a specific year with non tuberculosis cases who were alive until the end of that specific year based on birth year and sex. <strong>Matched non-tuberculosis cases were assigned an index date that corresponded to the first date of tuberculosis diagnosis of the matched tuberculosis survivors.</strong> If this matched non-tuberculosis case met the same exclusion criteria (ie, nonparticipants in health screening program within 2 year of index date, health screening data unavailable, ischemic stroke before index date, death, or ischemic stroke within 1 year after index date), they were excluded and new non-tuberculosis cases were matched based on the same exclusion criteria. Finally, 72 863 matched non-tuberculosis cases were included (Figure 1).</p>
</blockquote>
<p>결핵환자에 대해 비결핵 환자군을 매칭한 방식입니다. 비결핵 환자들의 Index date는 매칭된 결핵환자의 결핵 진단 날짜를 그대로 갖고옵니다. 매칭 이후에 제외된다면 또 다시 비결핵 환자군에서 매칭한 후 이 과정을 반복합니다.</p>
<blockquote class="blockquote">
<p><strong>[예시 논문 3] Strongman, Helen et al.&nbsp;(The Lancet, 2019)</strong><sup>3</sup></p>
<p>Cancer survivors entered the study <strong>1 year after diagnosis (index date)</strong> and were matched on age (±3 years), sex, and general practice with up to five controls with no history of cancer and at least 24 months of continuous preceding follow-up on the index date of the matched cancer survivor (mirroring the requirement on cancer survivors to have 1 year of follow-up before and after cancer diagnosis to enter the study on the index date). If a control went on to receive an incident cancer diagnosis during follow-up, they were no longer available as a control at that time but could then contribute as a site-specific cancer survivor (if the cancer was at one of the 20 sites of interest) with their own set of matched controls.</p>
</blockquote>
<p>암 생존자는 진단 1년 후(index date) 연구에 진입했으며, 나이(±3년), 성별, 병원이 동일하고 암 병력이 없는 최대 5명의 대조군과 매칭되었습니다. 대조군은 매칭된 암 생존자의 <strong>index date에 최소 24개월의 연속적인 추적 관찰 기간</strong>을 가지고 있어야 했습니다. 만약 대조군에 속한 사람이 추적 기간 동안 암 진단을 받게 된다면, 암 생존자로 바뀌어 그 자체의 매칭된 대조군을 갖게 되는 방식입니다.</p>
</section><section id="방법-2-대조군-고유의-특정-날짜-사용" class="level3"><h3 class="anchored" data-anchor-id="방법-2-대조군-고유의-특정-날짜-사용">방법 2: 대조군 고유의 특정 날짜 사용</h3>
<p>두 번째 방법은 치료군의 날짜가 아니라, <strong>대조군 개인이 가진 고유의 특정 날짜</strong>를 Index Date로 사용하는 것입니다. 예를 들어 데이터베이스 상의 마지막 기록 날짜 또는 특정 건강검진일 등을 사용할 수 있습니다.</p>
<ul>
<li>
<strong>장점</strong>:
<ul>
<li>대조군에서 비롯된 실제 날짜를 부여할 수 있습니다.</li>
</ul>
</li>
<li>
<strong>단점</strong>:
<ul>
<li>선택한 날짜의 타당성이 중요합니다. 마지막 방문일이 질병 진단일과 임상적으로 동등한 의미인지 생각해봐야 합니다.</li>
<li>치료군과 대조군의 Index Date가 달력상 다른 시점에 분포하게 되어, 시점에 따른 편향이 발생할 수 있습니다. (Immortal Time Bias 관련 블로그 글 읽기!)</li>
</ul>
</li>
</ul>
<blockquote class="blockquote">
<p><strong>[예시 논문 1] Nedelec, Thomas et al.&nbsp;(The Lancet Digital Health, 2022)</strong><sup>4</sup></p>
<p>The control individuals were then matched with patients with Alzheimer’s disease for age (SD 1 year) <strong>at last record in the database</strong> and sex.</p>
</blockquote>
<p>대조군은 <strong>데이터베이스의 마지막 기록 날짜(last record in the database)</strong>에서의 연령(±1년)과 성별을 기준으로 알츠하이머 환자군과 매칭되었습니다.</p>
</section><section id="방법-3-치료군-index-date-pool에서-무작위-할당-후-매칭" class="level3"><h3 class="anchored" data-anchor-id="방법-3-치료군-index-date-pool에서-무작위-할당-후-매칭">방법 3: 치료군 Index Date Pool에서 무작위 할당 후 매칭</h3>
<p>세 번째 방법은 조금 더 정교한 접근 방식입니다.</p>
<ol type="1">
<li>모든 <strong>치료군의 Index Date(예: 수술일) 목록(Pool)을 만듭니다.</strong>
</li>
<li>이 날짜들을 잠재적 대조군 전체에게 <strong>무작위(random)로 할당</strong> 합니다.</li>
<li>각 대조군은 무작위 할당된 Index Date (Pseudodate)를 기준으로 연구 포함/제외 기준을 만족하는지 확인합니다.</li>
<li>이 과정을 통해 수술 자격이 있었지만 받지 않은 대조군 풀이 생성됩니다.</li>
<li>이 풀에서 치료군과 1:N PSM 또는 Exact Matching을 수행합니다.</li>
</ol>
<ul>
<li>
<strong>장점</strong>:
<ul>
<li>대조군이 특정 시점에 치료(수술)를 받을 자격이 있었음을 시뮬레이션할 수 있습니다.</li>
<li>대조군 전체의 Index Date 분포가 치료군 전체의 Index Date 분포와 동일하게 되어, 시간에 따른 편향을 효과적으로 제어할 수 있습니다.</li>
</ul>
</li>
<li>
<strong>단점</strong>:
<ul>
<li>구현이 다른 방식에 비해 복잡합니다.</li>
</ul>
</li>
</ul>
<blockquote class="blockquote">
<p><strong>[예시 논문 1] Aminian A, et al.&nbsp;(JAMA, 2022)</strong><sup>5</sup></p>
<p>To create a comparable control group, <strong>dates for bariatric surgery were randomly assigned to a pool of 666,451 patients with a body mass index</strong> (BMI; calculated as weight in kilograms divided by height in meters squared) of 35 or greater. Patients who had not undergone bariatric surgery were then removed from the pool if they failed to meet inclusion criteria on the assigned date, at which point the patients could be seen as potentially eligible for bariatric surgery. Using this algorithm, 128,119 comparable patients who had not undergone surgery were identified to be considered for matching. <strong>With propensity matching of each patient who underwent bariatric surgery to 5 patients who had not undergone surgery</strong> (nonsurgical control), 5053 patients in the bariatric surgery group and 25,265 matched patients in the nonsurgical control group were enrolled in the study. Doubly robust estimation combining the propensity score and outcome regression was used. <strong>Each patient who underwent bariatric surgery was matched with a propensity score by the nearest-neighbor method to 5 patients who did not undergo bariatric surgery</strong> (nonsurgical control), using a logistic regression model based on 10 a priori–identified potential confounders. The matching variables included the index date, age, sex, race (which was obtained from the EHR based on patient self-report using fixed categories and was classified as Black, White, or other), BMI on the index date (35-39.9, 40-44.9, 45-49.9, 50-54.9, 55-59.9, or 60-80), smoking history (categorized as never, former, or current), presence of type 2 diabetes, Elixhauser Comorbidity Index, Charlson Comorbidity Index, and state of residence (Florida, Ohio, or other US state).</p>
</blockquote>
<p>대조군을 만들기 위해 치료군의 비만대사수술 날짜들을 BMI 35 이상인 666,451명의 수술 받지 않은 환자 풀에 <strong>무작위로 할당</strong>했습니다. 수술을 받지 않은 환자들은 이 할당된 날짜(Pseudodate) 에 포함 기준을 충족하지 못하면 풀에서 제거되고, 모든 조건이 충족 된다면 그 매칭된 대조군의 Index date가 바로 그 Pseudodate가 됩니다. 이 알고리즘을 사용하여 매칭을 위해 고려할 수 있는 128,119명의 비교 가능한 비수술 환자를 식별했습니다. 1:5 Propensity Score 매칭을 통해 수술 그룹 5,053명과 비수술 대조군 25,265명이 연구에 등록되었습니다.</p>
</section></section></section><section id="conclusion" class="level1"><h1>Conclusion</h1>
<p>이 글에서는 코호트 연구와 환자-대조군 연구의 주요 차이점을 비교하며 연구 설계의 중요성을 확인했습니다. 코호트 연구를 성공적으로 수행하고 결과의 타당성을 확보하기 위해 꼭 다뤄야 할 주제는 바로 대조군의 Index date 설정입니다.</p>
<p>코호트 연구는 노출 시점에서 미래의 결과를 추적하지만, 대조군이 노출군과 동일한 방식으로 추적되려면 노출군에게 질병이 진단된 시점과 유사하게 설정되어야 합니다. 가상 노출 시작일 또는 가상의 질병 발생 위험에 노출된 시점을 정확히 정의해야 합니다. 이 Index date를 잘못 설정하면 선택 편향으로 이어질 수 있습니다.</p>
<p>따라서 코호트 연구의 최종 해석을 내릴 때, 연구자는 대조군의 Index date 설정 방식이 연구의 목적과 노출-결과 관계의 시간성을 얼마나 정확하게 반영했는지 확인해야 합니다. 적절한 Index date 정의는 코호트 연구의 타당성을 지키는 증요한 부분입니다.</p>


</section><div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>
<ol>
<li id="fn1"><p>Forbes, Harriet, et al.&nbsp;“Early, medium and long-term mental health in cancer survivors compared with cancer-free comparators: matched cohort study using linked UK electronic health records.” EClinicalMedicine 76 (2024).↩︎</p></li>
<li id="fn2"><p>Lee, Han Rim, et al.&nbsp;“Tuberculosis and risk of ischemic stroke: a nationwide cohort study.” Stroke 53.11 (2022): 3401-3409.↩︎</p></li>
<li id="fn3"><p>Strongman, Helen, et al.&nbsp;“Medium and long-term risks of specific cardiovascular diseases in survivors of 20 adult cancers: a population-based cohort study using multiple linked UK electronic health records databases.” The Lancet 394.10203 (2019): 1041-1054.↩︎</p></li>
<li id="fn4"><p>Nedelec, Thomas, et al.&nbsp;“Identifying health conditions associated with Alzheimer’s disease up to 15 years before diagnosis: an agnostic study of French and British health records.” The Lancet Digital Health 4.3 (2022): e169-e178.↩︎</p></li>
<li id="fn5"><p>Aminian, Ali, et al.&nbsp;“Association of bariatric surgery with cancer risk and mortality in adults with obesity.” Jama 327.24 (2022): 2423-2433.↩︎</p></li>
</ol></section><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hahn2025,
  author = {Hahn, Wonbin},
  title = {코호트 {연구의} {핵심,} {대조군의} {Index} Date {설정}},
  date = {2025-11-19},
  url = {https://blog.zarathu.com/posts/2025-11-18-control_index/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hahn2025" class="csl-entry quarto-appendix-citeas">
Hahn, Wonbin. 2025. <span>“코호트 연구의 핵심, 대조군의 Index Date
설정.”</span> November 19, 2025. <a href="https://blog.zarathu.com/posts/2025-11-18-control_index/">https://blog.zarathu.com/posts/2025-11-18-control_index/</a>.
</div></div></section></div> ]]></description>
  <category>review</category>
  <guid>https://blog.zarathu.com/posts/2025-11-18-control_index/</guid>
  <pubDate>Wed, 19 Nov 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-11-18-control_index/img/logo.png" medium="image" type="image/png" height="144" width="144"/>
</item>
<item>
  <title>AFROC vs. JAFROC: MRMC 연구의 통계 분석</title>
  <dc:creator>Wonbin Hahn</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-10-28-jafroc/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="introduction" class="level1"><h1>Introduction</h1>
<p>이 글은 의료 영상 진단 기기(예: AI)의 성능을 평가할 때 사용되는 두 가지 핵심 방법론, <strong>AFROC</strong>와 <strong>JAFROC</strong>을 비교하고 JAFROC의 원리에 대해 설명합니다.</p>
<section id="mrmc-연구란" class="level2"><h2 class="anchored" data-anchor-id="mrmc-연구란">MRMC 연구란?</h2>
<p><strong>MRMC (Multi-Reader, Multi-Case)</strong> 연구는 <strong>여러 명의 판독자(Multi-Reader)</strong>가 <strong>여러 개의 증례(Multi-Case)</strong>를 보고 진단 성능을 평가하는 연구 설계입니다.</p>
<p>예를 들어, 10명의 영상의학과 의사(판독자)가 100명의 환자(증례) 데이터를 A 방법(예: AI 없음)과 B 방법(예: AI 사용)으로 모두 판독하는 것입니다.</p>
</section><section id="mrmc는-왜-분석이-까다로운가" class="level2"><h2 class="anchored" data-anchor-id="mrmc는-왜-분석이-까다로운가">MRMC는 왜 분석이 까다로운가?</h2>
<p>MRMC 연구의 목표는 “A 방법과 B 방법 중 어느 것이 더 우수한가?”를 통계적으로 검증하는 것입니다. 하지만 여기에는 여러 <strong>변동성(Variability)</strong> 요인이 복합적으로 얽혀있습니다.</p>
<ol type="1">
<li>
<strong>증례 변동성 (Case Variability)</strong>: 어떤 증례의 환자 데이터는 진단이 쉽고, 어떤 데이터는 어렵습니다.</li>
<li>
<strong>독자 변동성 (Reader Variability)</strong>: 어떤 판독자는 전반적으로 실력이 좋고, 어떤 판독자는 안 좋습니다.</li>
<li>
<strong>독자-처치 상호작용 (Reader-Treatment Interaction)</strong>: 어떤 의사는 유독 A 방법(AI 없음)에 익숙하고, 다른 의사는 B 방법(AI 사용)에서만 성능이 크게 향상될 수 있습니다.</li>
</ol>
<p>올바른 통계 분석은 이 모든 변동성 요인을 고려하여 “평균적으로 B 방법이 A 방법보다 우수하다”고 말할 수 있는지 검증해야 합니다.</p>
</section><section id="mrmc-연구의-성능-평가" class="level2"><h2 class="anchored" data-anchor-id="mrmc-연구의-성능-평가">MRMC 연구의 성능 평가</h2>
<p>진단 성능을 평가하는 지표로 ROC AUC가 널리 쓰입니다. 하지만 이는 증례 당 “정상/비정상” 여부만 판단합니다. 이외에도 <strong>LROC, FROC, AFROC, JAFROC</strong> 등이 있습니다.</p>
<p>이 글은 <strong>ROC와 FROC의 한계</strong>를 짚고, <strong>AFROC</strong>와 <strong>JAFROC</strong>가 MRMC에서 어떻게 쓰이는지, 그리고 둘의 차이를 설명합니다.</p>
</section></section><section id="성능-지표-roc-에서-jafroc까지지" class="level1"><h1>성능 지표: ROC 에서 JAFROC까지지</h1>
<section id="roc-receiver-operating-characteristic" class="level2"><h2 class="anchored" data-anchor-id="roc-receiver-operating-characteristic">ROC (Receiver Operating Characteristic)</h2>
<p><strong>ROC</strong> 분석은 <strong>병변 유무(검출)</strong> 성능을 평가합니다. 하지만 ROC는 <strong>병변의 위치(Localization)</strong>를 반영하지 못합니다.</p>
<p>이러한 ROC의 한계를 보완한 분석 곡선이 몇 가지 있습니다.</p>
</section><section id="froc-free-response-roc" class="level2"><h2 class="anchored" data-anchor-id="froc-free-response-roc">FROC (Free-Response ROC)</h2>
<p><strong>FROC</strong>는 병변의 개수에 제약을 두지 않고 평가를 진행한 곡선입니다. 즉, ROC와 다르게 병변 수가 0, 1, 2… 이 될 수 있습니다.</p>
<ul>
<li>
<p><strong>x축:</strong> <strong>NLF (Non-lesion Localization Fraction)</strong> - 이미지당 <em>평균</em> 오탐(비병변 위치 표식) 수 [0, <img src="https://latex.codecogs.com/png.latex?Inf">]</p>
<p>average number of false-positive localizations per image (FP marks per image)</p>
</li>
<li>
<p><strong>y축:</strong> <strong>LLF (Lesion Localization Fraction)</strong> -<em>병변 단위</em> 민감도 [0,1]</p>
<p>proportion of lesions correctly localized (per-lesion sensitivity)</p>
</li>
</ul>
<p>하지만 FROC의 x축은 상한이 없어 병변의 수에 따라 곡선 범위가 달라집니다.</p>
<p>결과적으로 <strong>AUC를 일관되게 정의하고 비교하기 어렵습니다.</strong></p>
</section><section id="afroc-alternative-free-response-roc" class="level2"><h2 class="anchored" data-anchor-id="afroc-alternative-free-response-roc">AFROC (Alternative Free-Response ROC)</h2>
<p><strong>AFROC</strong>는 “Free-Response”라는 이름처럼, 한 증례(예: Mammography 이미지 한 장) 안에 <strong>여러 개의 병변이 있을 때, 그 병변의 위치까지 올바르게 찾아냈는지</strong>를 반영하는 성능 지표입니다.</p>
<p><strong>AFROC</strong>는 FROC의 한계를 보완합니다. 마찬가지로 병변의 개수에 제약을 두지 않지만, x축을 다르게 정의하여 [0,1] 범위로 고정시킵니다.</p>
<ul>
<li><p><strong>x축:</strong> <strong>FPF (False Positive Fraction)</strong> — <strong>정상 이미지</strong> 중 임계값을 넘는 오탐이 <strong>하나라도</strong> 있는 비율 [0,1]</p></li>
<li><p><strong>y축:</strong> <strong>LLF (Lesion Localization Fraction)</strong> — <em>병변 단위</em> 민감도 [0,1]</p></li>
</ul>
<p>두 축이 모두 [0,1] 범위로 고정되므로 <strong>AUC 계산이 안정적</strong>이고 <strong>곡선 간 비교가 용이</strong>합니다.</p>
<p>AFROC 곡선의 AUC(곡선 아래 면적)를 <strong>FOM (Figure of Merit)</strong>이라고 부르며, 이것이 JAFROC 분석에서 사용할 핵심 성능 지표가 됩니다.</p>
</section><section id="jafroc-afroc-auc-mrmc-추론-레이어" class="level2"><h2 class="anchored" data-anchor-id="jafroc-afroc-auc-mrmc-추론-레이어">JAFROC: AFROC AUC + MRMC 추론 레이어</h2>
<p><strong>JAFROC</strong>는 <strong>AFROC와 동일한 곡선과 동일한 AUC(FOM)</strong>를 사용하되, <strong>Case-level Jackknife resampling</strong>와 <strong>OR/ORH형 MRMC분석</strong>을 결합해 아래를 제공합니다.</p>
<ul>
<li>
<strong>표준오차(SE)</strong>, <strong>신뢰구간(CI)</strong>, <strong>유의성 검정(p-value)</strong>
</li>
<li>
<strong>상관구조 반영:</strong> 같은 독자·다른 처치, 같은 처치·다른 독자, 서로 다른 독자·처치 등 <strong>공유 케이스로 인한 공분산</strong> 반영</li>
</ul>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p><strong>중요:</strong> JAFROC는 “새로운 곡선” 이 아닙니다.<br><strong>AFROC AUC(FOM)는 동일</strong>하며, JAFROC는 그 <strong>동일한 FOM</strong>에 대해<br><strong>올바른 분산/상관 추정과 MRMC 검정</strong>을 얹는 통계적 분석 프레임워크입니다.</p>
</div>
</div>
</section></section><section id="afroc-vs.-jafroc-mrmc-관점에서" class="level1"><h1>AFROC vs.&nbsp;JAFROC (MRMC 관점에서)</h1>
<p>MRMC 연구의 목표는 여러 처치(예: A 방법, B 방법)의 진단 성능을 비교하는 것입니다.</p>
<p>가장 단순한 방법은 각 독자/처치별로 <strong>AFROC AUC(FOM)</strong>를 구한 뒤, 이 점수들을 <strong>단순 평균</strong>하여 비교하는 것입니다.</p>
<p>하지만 이 ‘단순 평균’ 방식은 MRMC의 <strong>핵심 변동성 3가지</strong>를 제대로 반영하지 못하며, 이로 인해 <strong>표본 오차(SE)를 실제보다 낮게 추정</strong>하는 한계가 있습니다.</p>
<section id="jafroc가-필요한-이유-단순-afroc의-3가지-한계" class="level2"><h2 class="anchored" data-anchor-id="jafroc가-필요한-이유-단순-afroc의-3가지-한계">JAFROC가 필요한 이유: 단순 AFROC의 3가지 한계</h2>
<p>AFROC는 아래 3가지 변동성을 무시합니다.</p>
<ol type="1">
<li>
<strong>판독자 간 변동성 (Reader Variability)</strong>
<ul>
<li>
<strong>문제점:</strong> 어떤 의사는 전반적으로 성능이 높고(예: 항상 90점), 어떤 의사는 낮습니다(예: 항상 70점). 이 ’의사’라는 요인 자체가 거대한 변동성의 원천입니다.</li>
</ul>
</li>
<li>
<strong>판독자-처치 간 상호작용 (Reader-Treatment Interaction)</strong>
<ul>
<li>
<strong>문제점:</strong> A 의사는 1번 처치(AI 없음)에 익숙해 80점을 받았지만, 2번 처치(AI 사용)에는 익숙하지 않아 75점을 받을 수 있습니다. 반면 B 의사는 1번 처치 70점, 2번 처치 85점을 받을 수 있습니다. 이처럼 <strong>’특정 의사’와 ’특정 처치’가 만났을 때</strong> 발생하는 변동성을 반영해야 합니다.</li>
</ul>
</li>
<li>
<strong>증례 유래 공분산 (Case-derived Covariance)</strong>
<ul>
<li>
<strong>문제점:</strong> MRMC 연구에서는 <strong>‘같은 독자’</strong>가 <strong>‘같은 증례(환자)’</strong>를 여러 처치(A, B)로 모두 판독합니다.</li>
<li>만약 어떤 증례가 매우 어렵다면(데이터가 안 좋다면), 그 의사는 A 방법에서도 낮은 점수를, B 방법에서도 낮은 점수를 줄 것입니다.</li>
<li>이처럼 <strong>“같은 증례”</strong>를 공유하기 때문에 발생하는 처치 간 점수(FOM)의 상관관계(공분산)를 반드시 고려해야 합니다.</li>
</ul>
</li>
</ol>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
</div>
<div class="callout-body-container callout-body">
<p><strong>Q: Case-level Bootstrap은 왜 안되나요?</strong></p>
<p>Bootstrap resampling은 (3)번 문제(증례 변동성)는 일부 반영할 수 있지만, (1)번과 (2)번 (판독자 및 판독자-처치 상호작용) 변동성을 제대로 다루지 못합니다. 따라서 여전히 불완전한 방법이며 MRMC의 표준으로 JAFROC를 사용합니다.</p>
</div>
</div>
</section><section id="jafroc-3가지-한계를-모두-해결하는-프레임워크" class="level2"><h2 class="anchored" data-anchor-id="jafroc-3가지-한계를-모두-해결하는-프레임워크">JAFROC: 3가지 한계를 모두 해결하는 프레임워크</h2>
<p><strong>JAFROC</strong>는 이 모든 문제를 해결하기 위해 설계된 <strong>통계적 분석 프레임워크</strong>입니다.</p>
<p>JAFROC는 AFROC와 <strong>동일한 곡선, 동일한 AUC(FOM)</strong>를 사용한다는 점이 중요합니다. JAFROC는 그저 이 FOM 점수들을 통계적으로 올바르게 분석하는 ’방법론’의 이름입니다.</p>
<ol type="1">
<li>
<strong>[Solution for 3]</strong> <code>Case-level Jackknife</code> 방식을 사용하여 <strong>증례 유래 공분산</strong>을 정확하게 추정합니다.</li>
<li>
<strong>[Solution for 1, 2]</strong> <code>OR/ORH 모델</code>이라는 MRMC 통계 모델을 사용하여 <strong>독자 변동성</strong>과 <strong>독자-처치 상호작용</strong>을 모두 반영합니다.</li>
</ol>
<section id="요약" class="level3"><h3 class="anchored" data-anchor-id="요약">요약</h3>
<ul>
<li>
<strong>AFROC</strong>: 성능 곡선(Plot)과 그 AUC(FOM)라는 <strong>‘성능 지표’</strong>를 제공합니다.</li>
<li>
<strong>JAFROC</strong>: AFROC AUC라는 <strong>‘성능 지표’</strong>는 그대로 사용하되, Jackknife와 OR/ORH 모델을 결합하여 MRMC의 모든 변동성을 올바르게 반영하는 <strong>‘통계 분석 방법론’</strong>입니다.</li>
</ul>
<p>따라서 JAFROC는 AFROC이 제공하지 못하는 <strong>표준오차(SE), 신뢰구간(CI), 그리고 유의성 검정(p-value)</strong>을 제공합니다.</p>
</section></section><section id="jafroc-작동-원리" class="level2"><h2 class="anchored" data-anchor-id="jafroc-작동-원리">JAFROC 작동 원리</h2>
<p>JAFROC의 계산 과정은 다음과 같습니다.</p>
<section id="case-level-jackknife-resampling" class="level3"><h3 class="anchored" data-anchor-id="case-level-jackknife-resampling">(1) Case-level Jackknife Resampling</h3>
<p>증례가 K개일 때 처치 i, 판독자 j의 Full FOM <img src="https://latex.codecogs.com/png.latex?%7B%5Ctheta%7D_%7Bij%7D">와 증례 k를 제거한 Delete-1-case FOM <img src="https://latex.codecogs.com/png.latex?%7B%5Ctheta%7D_%7Bij(-k)%7D">를 계산합니다.</p>
<ul>
<li>
<strong>Full FOM:</strong> <img src="https://latex.codecogs.com/png.latex?%5C%20%5Chat%7B%5Ctheta%7D_%7Bij%7D">
</li>
<li>
<strong>Delete-1-case FOM:</strong> <img src="https://latex.codecogs.com/png.latex?%5C%20%5Chat%7B%5Ctheta%7D_%7Bij(-k)%7D"> (케이스 <img src="https://latex.codecogs.com/png.latex?k"> 제거)</li>
</ul>
<p>다음과 같이 Jackknife Pseudovalue <img src="https://latex.codecogs.com/png.latex?P_%7Bijk%7D"> 를 구합니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_%7Bijk%7D%20%5C;=%5C;%20K%5C,%5Chat%7B%5Ctheta%7D_%7Bij%7D%20%5C;-%5C;%20(K-1)%5C,%5Chat%7B%5Ctheta%7D_%7Bij(-k)%7D%0A"></p>
<p>Delete-1-Case FOM을 계산할 때 증례 k를 제거하면 그 케이스의 <strong>모든 병변 마크</strong>가 한꺼번에 사라집니다. Jackknife Pseudovalue는 Delete-1-Case FOM을 이용하여 구하기 때문에 <img src="https://latex.codecogs.com/png.latex?P_%7Bijk%7D">가 증례 k에 포함된 <strong>병변 클러스터 전체의 기여를 반영</strong>합니다.</p>
<p>한 증례에 포함된 병변 클러스터가 같이 이동하기 때문에 분산을 구하면 다음과 같이 <strong>각 병변의 분산 + 병변–병변 공분산이 포함됩니다.</strong></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathrm%7BVar%7D%5C!%5CBig(%5Csum_%7Bm=1%7D%5E%7BL_k%7D%5Cpsi(Y_%7Bk,m%7D)%5CBig)%0A=%5Csum_%7Bm=1%7D%5E%7BL_k%7D%5Cmathrm%7BVar%7D%5Cbig(%5Cpsi(Y_%7Bk,m%7D)%5Cbig)%0A+2%5C!%5C!%5Csum_%7B1%5Cle%20m%3Cm'%5Cle%20L_k%7D%5C!%5C!%5Cmathrm%7BCov%7D%5C!%5Cbig(%5Cpsi(Y_%7Bk,m%7D),%5Cpsi(Y_%7Bk,m'%7D)%5Cbig)%0A"></p>
<p><strong>즉, 케이스 내 병변-병변 상관(intra-case, intra-lesion correlation)</strong>이 자동 반영됩니다.</p>
</section><section id="case-level-jackknife-covariance-components" class="level3"><h3 class="anchored" data-anchor-id="case-level-jackknife-covariance-components">(2) Case-level jackknife covariance components</h3>
<ul>
<li>
<strong>Var</strong> (same treatment, same reader):<br><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D%20%5C;%5Cpropto%5C;%20%5Csum_%7Bk=1%7D%5E%7BK%7D%5C,%20%5Ctilde%20P_%7Bijk%7D%5E%7B%5C,2%7D%0A">
</li>
<li>
<strong>Cov</strong><img src="https://latex.codecogs.com/png.latex?_1"> (different treatments, same reader):<br><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Cmathrm%7BCov%7D%7D_1%20%5C;%5Cpropto%5C;%20%5Csum_%7Bi%5Cne%20i'%7D%5Csum_%7Bk=1%7D%5E%7BK%7D%5C,%20%5Ctilde%20P_%7Bijk%7D%5C,%5Ctilde%20P_%7Bi'jk%7D%0A">
</li>
<li>
<strong>Cov</strong><img src="https://latex.codecogs.com/png.latex?_2"> (same treatment, different readers):<br><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Cmathrm%7BCov%7D%7D_2%20%5C;%5Cpropto%5C;%20%5Csum_%7Bj%5Cne%20j'%7D%5Csum_%7Bk=1%7D%5E%7BK%7D%5C,%20%5Ctilde%20P_%7Bijk%7D%5C,%5Ctilde%20P_%7Bij'k%7D%0A">
</li>
<li>
<strong>Cov</strong><img src="https://latex.codecogs.com/png.latex?_3"> (different treatments, different readers):<br><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Cmathrm%7BCov%7D%7D_3%20%5C;%5Cpropto%5C;%20%5Csum_%7Bi%5Cne%20i'%7D%5Csum_%7Bj%5Cne%20j'%7D%5Csum_%7Bk=1%7D%5E%7BK%7D%5C,%20%5Ctilde%20P_%7Bijk%7D%5C,%5Ctilde%20P_%7Bi'j'k%7D%0A">
</li>
</ul></section><section id="ororhmrmc-독자-무작위-효과" class="level3"><h3 class="anchored" data-anchor-id="ororhmrmc-독자-무작위-효과">(3) OR/ORH(MRMC): 독자 무작위 효과</h3>
<ul>
<li>
<strong>독자 효과</strong>: <img src="https://latex.codecogs.com/png.latex?R_j%20%5Csim%20%5Cmathcal%7BN%7D(0,%5Csigma_R%5E2)"><br>
</li>
<li>
<strong>독자×처치 효과</strong>: <img src="https://latex.codecogs.com/png.latex?%5Ctau%20R_%7Bij%7D%20%5Csim%20%5Cmathcal%7BN%7D(0,%5Csigma_%7B%5Ctau%20R%7D%5E2)">
</li>
</ul>
<p>OR/ORH는 <strong>Obuchowski-Rockette (OR) 모델</strong> (또는 그 확장형인 Hillis(H) 모델)을 의미하며, MRMC 연구 분석을 위해 특별히 설계된 <strong>무작위 효과 모델 (Random-Effects Model)</strong>입니다.</p>
<p>이 모델이 JAFROC의 핵심인 이유는 앞서 언급한 <strong>(1) 독자 간 변동성</strong>과 <strong>(2) 독자-처치 간 상호작용</strong>을 통계적으로 처리하기 때문입니다.</p>
<p>JAFROC는 (1), (2)에서 잭나이프로 얻은 <strong>Case level 공분산 성분들</strong>(<img src="https://latex.codecogs.com/png.latex?%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D,%20%5Cwidehat%7B%5Cmathrm%7BCov%7D%7D_1,%20%5Cdots">)과 (3)의 OR/ORH 모델로 추정한 <strong>Reader-level 분산 성분들</strong>(<img src="https://latex.codecogs.com/png.latex?%5Csigma_R%5E2,%20%5Csigma_%7B%5Ctau%20R%7D%5E2">)을 통계적으로 모두 결합합니다.</p>
<p>이 과정을 통해, 우리가 진짜 알고 싶은 <strong>’독자-평균 처치 차이(</strong><img src="https://latex.codecogs.com/png.latex?%5Cbar%20d">)’의 <strong>총 분산(Total Variance)</strong>을 정확하게 계산해내며, 이를 바탕으로 신뢰할 수 있는 <strong>SE, CI, 그리고 F-검정</strong>을 제공하는 것입니다.</p>
</section><section id="최종-가설-검정-및-결론" class="level3"><h3 class="anchored" data-anchor-id="최종-가설-검정-및-결론">(4) 최종 가설 검정 및 결론</h3>
<p>(1), (2), (3)에서 계산된 분산-공분산 성분과 무작위 효과 모델(OR/ORH)은 MRMC 연구의 핵심 질문에 답하기 위해 사용됩니다.</p>
<p>JAFROC MRMC 분석의 최종 목표는 <strong>“여러 처치(Modality) 간에 독자-평균(reader-averaged) 성능(FOM, 즉 AFROC AUC)에 차이가 없다”</strong>는 귀무가설(Null Hypothesis)을 검정하는 것입니다.</p>
<ul>
<li>
<strong>귀무가설 (</strong><img src="https://latex.codecogs.com/png.latex?H_0">): <img src="https://latex.codecogs.com/png.latex?%5Cmu_1%20=%20%5Cmu_2%20=%20%5Cdots%20=%20%5Cmu_I">
<ul>
<li>(여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmu_i">는 처치 <img src="https://latex.codecogs.com/png.latex?i">에 대한 모집단의 평균 FOM입니다.)</li>
</ul>
</li>
<li>
<strong>대립가설 (</strong><img src="https://latex.codecogs.com/png.latex?H_A">): 적어도 하나의 처치에서 평균 FOM이 다릅니다.</li>
</ul>
<p>(3)에서 언급된 <strong>F-검정</strong>이 바로 이 귀무가설을 검정합니다. OR/ORH 모델은 (2)에서 잭나이프로 추정한 <strong>증례 유래 공분산(</strong><img src="https://latex.codecogs.com/png.latex?%5Cwidehat%7B%5Cmathrm%7BCov%7D%7D_1,%20%5Cwidehat%7B%5Cmathrm%7BCov%7D%7D_2,%20%5Cdots">)과 (3)의 <strong>독자 및 독자-처치 상호작용 변동성(</strong><img src="https://latex.codecogs.com/png.latex?%5Csigma_R%5E2,%20%5Csigma_%7B%5Ctau%20R%7D%5E2">)을 모두 반영하여 F-통계량의 분모(오차항)를 구성합니다.</p>
</section></section><section id="r-코드를-이용한-실습" class="level2"><h2 class="anchored" data-anchor-id="r-코드를-이용한-실습">R 코드를 이용한 실습</h2>
<p><code>RJafroc</code> 패키지를 설치하고 불러옵니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://dpc10ster.github.io/RJafroc/">RJafroc</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>패키지 안에 있는 <code>dataset01</code>을 불러옵니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># MRMC 샘플 데이터(2 처치, 5 독자, 185 증례)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/data.html">data</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dataset01</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><code>UtilFigureOfMerit</code> 함수를 사용하면 <strong>개별 FOM(즉, 판독자별 AFROC AUC)</strong>을 구할 수 있습니다. 아래는 각 처치(trtBT, trtDM) 별로 각 판독자(rdr)의 AUC 결과입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># dataset01 (2 처치, 5 독자)의 개별 FOM 계산</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">UtilFigureOfMerit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  dataset <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dataset01</span>,</span>
<span>  FOM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AFROC"</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>           rdr1      rdr2      rdr3      rdr4      rdr5
trtBT 0.7591557 0.8375000 0.8131031 0.8071272 0.8277961
trtDM 0.6406798 0.7034539 0.7359101 0.7694079 0.6805921</code></pre>
</div>
</div>
<p>마찬가지로 dataset01 을 불러와서 <code>St</code> 함수를 통해 JAFROC의 <strong>최종 F-검정(p-value)</strong>을 한 번에 보여줍니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># JAFROC MRMC 분석 수행</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_result</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">St</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  dataset <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dataset01</span>,</span>
<span>  FOM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AFROC"</span>,</span>
<span>  method <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"OR"</span> </span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 분석 결과(ANOVA 테이블) 출력</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 이 결과에 우리가 찾던 F-검정, p-value가 포함됩니다.</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/print.html">print</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_result</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>$FOMs
$FOMs$foms
           rdr1      rdr2      rdr3      rdr4      rdr5
trtBT 0.7591557 0.8375000 0.8131031 0.8071272 0.8277961
trtDM 0.6406798 0.7034539 0.7359101 0.7694079 0.6805921

$FOMs$trtMeans
       Estimate
trtBT 0.8089364
trtDM 0.7060088

$FOMs$trtMeanDiffs
             Estimate
trtBT-trtDM 0.1029276


$ANOVA
$ANOVA$TRanova
            SS DF          MS
T  0.026485243  1 0.026485243
R  0.009461813  4 0.002365453
TR 0.004042448  4 0.001010612

$ANOVA$VarCom
         Estimates      Rhos
VarR  0.0006501141        NA
VarTR 0.0004842873        NA
Cov1  0.0002792523 0.2565054
Cov2  0.0005350491 0.4914658
Cov3  0.0002519457 0.2314231
Var   0.0010886803        NA

$ANOVA$IndividualTrt
      DF   msREachTrt   varEachTrt  cov2EachTrt
trtBT  4 0.0009175795 0.0009568316 0.0005106125
trtDM  4 0.0024584857 0.0012205289 0.0005594856

$ANOVA$IndividualRdr
     DF   msTEachRdr   varEachRdr  cov1EachRdr
rdr1  1 0.0070182667 0.0013828510 0.0001481347
rdr2  1 0.0089841721 0.0009578877 0.0003237347
rdr3  1 0.0029793783 0.0011344453 0.0003160499
rdr4  1 0.0007113727 0.0008835103 0.0002868516
rdr5  1 0.0108345011 0.0010847069 0.0003214908


$RRRC
$RRRC$FTests
                DF          MS    FStat           p
Treatment  1.00000 0.026485243 10.91667 0.003093432
Error     23.05254 0.002426129       NA          NA

$RRRC$ciDiffTrt
             Estimate     StdErr       DF        t       PrGTt    CILower
trtBT-trtDM 0.1029276 0.03115207 23.05254 3.304038 0.003093432 0.03849279
              CIUpper
trtBT-trtDM 0.1673625

$RRRC$ciAvgRdrEachTrt
       Estimate     StdErr       DF   CILower   CIUpper         Cov2
trtBT 0.8089364 0.02634632 57.22585 0.7561833 0.8616895 0.0005106125
trtDM 0.7060088 0.03242195 18.28189 0.6379680 0.7740496 0.0005594856</code></pre>
</div>
</div>
<p><code>St</code> 함수는 내부적으로</p>
<p>(1) 각 독자/처치별 FOM 계산 <img src="https://latex.codecogs.com/png.latex?%5Crightarrow"> (2) Jackknife <img src="https://latex.codecogs.com/png.latex?%5Crightarrow"> (3) OR/ORH 모델 적용</p>
<p><code>UtilFigureOfMerit</code> 함수는 이 중 (1)번 단계, 즉 각 독자(Reader)와 처치(Treatment) 조합별 AFROC AUC (FOM) 값(<img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Ctheta%7D_%7Bij%7D">)을 직접 계산해서 보여줍니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">UtilFigureOfMerit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>dataset <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dataset01</span>, FOM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AFROC"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>           rdr1      rdr2      rdr3      rdr4      rdr5
trtBT 0.7591557 0.8375000 0.8131031 0.8071272 0.8277961
trtDM 0.6406798 0.7034539 0.7359101 0.7694079 0.6805921</code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_result</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FOMs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">foms</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>           rdr1      rdr2      rdr3      rdr4      rdr5
trtBT 0.7591557 0.8375000 0.8131031 0.8071272 0.8277961
trtDM 0.6406798 0.7034539 0.7359101 0.7694079 0.6805921</code></pre>
</div>
</div>
<p><code>UtilFigureOfMerit</code> 함수의 결과와 <code>jafroc_result$FOMs$foms</code>를 비교해보면 값이 똑같습니다. 이를 통해 JAFROC는 AFROC와 똑같은 AUC를 쓴다는 것을 알 수 있습니다.</p>
<p><code>jafroc_result$FOMs$trtMeans</code>를 통해 전체 AUC는 독자별 AUC의 단순 평균이라는 사실도 알 수 있습니다. 아래 결과입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_result</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FOMs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">trtMeans</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>       Estimate
trtBT 0.8089364
trtDM 0.7060088</code></pre>
</div>
</div>
<p><code>jafroc_result$RRRC$FTests</code>가 RRRC (Random Reader, Random Case) 분석 방법에 대한 F-검정 결과를 보여줍니다. 아래 결과입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_result</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RRRC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FTests</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>                DF          MS    FStat           p
Treatment  1.00000 0.026485243 10.91667 0.003093432
Error     23.05254 0.002426129       NA          NA</code></pre>
</div>
</div>
<p>Treatment 행의 FStat (10.91667)와 p (0.003093432) 값이 바로 JAFROC 분석법으로 계산한 F-통계량과 p-value입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_plot</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">PlotEmpOpChrs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dataset01</span>, </span>
<span>              trts <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># two treatments</span></span>
<span>              rdrs <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># average over all readers for each treatment</span></span>
<span>              opChType <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AFROC"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">jafroc_plot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Plot</span></span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-10-28-jafroc/index_files/figure-html/unnamed-chunk-8-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>마지막으로 <code>PlotEmpOpChrs</code> 함수를 사용하여 판독자 평균 AFROC 곡선을 그릴 수 있습니다. 위 곡선 2개는 각 trt별 판독자 평균 AFROC 곡선입니다.</p>
<p>trts = list(1, 2)와 rdrs = list(1:5, 1:5)를 지정하면</p>
<ul>
<li><p>1번 처치에 대한 5명 독자의 평균 곡선</p></li>
<li><p>2번 처치에 대한 5명 독자의 평균 곡선</p></li>
</ul>
<p>이렇게 총 2개의 곡선을 비교하여 볼 수 있습니다.</p>
</section></section><section id="conclusion" class="level1"><h1>Conclusion</h1>
<p>지금까지 AFROC와 JAFROC의 결정적인 차이에 대해 알아봤습니다.</p>
<ul>
<li>
<strong>AFROC</strong>: 각 판독자의 AUC를 구해 단순 평균을 구할 수 있지만, MRMC의 복잡한 분산 구조(증례, 판독자, 상호작용)를 모두 반영한 통계적 검정(p-value)을 제공하지 못합니다.</li>
<li>
<strong>JAFROC</strong>: AFROC AUC를 성능지표(FOM)로 사용하되, 이를 MRMC의 모든 변동성 요인을 올바르게 고려하는 OR/ORH 통계 프레임워크에 결합합니다.</li>
</ul>
<p>결론적으로 JAFROC는 <strong>증례 내 병변 상관성</strong> (잭나이프로 해결)과 <strong>독자 간 변동성</strong> (OR/ORH 모델로 해결)을 모두 적절히 처리하여, 처치 간 성능 차이에 대한 <strong>p-value</strong>와 <strong>신뢰구간(CI)</strong>을 제공하는 MRMC 표준 분석법입니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hahn2025,
  author = {Hahn, Wonbin},
  title = {AFROC Vs. {JAFROC:} {MRMC} {연구의} {통계} {분석}},
  date = {2025-10-28},
  url = {https://blog.zarathu.com/posts/2025-10-28-jafroc/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hahn2025" class="csl-entry quarto-appendix-citeas">
Hahn, Wonbin. 2025. <span>“AFROC Vs. JAFROC: MRMC 연구의 통계
분석.”</span> October 28, 2025. <a href="https://blog.zarathu.com/posts/2025-10-28-jafroc/">https://blog.zarathu.com/posts/2025-10-28-jafroc/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-10-28-jafroc/</guid>
  <pubDate>Tue, 28 Oct 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-10-28-jafroc/img/logo.png" medium="image" type="image/png" height="144" width="144"/>
</item>
<item>
  <title>R 패키지 개발부터 CRAN 제출까지의 여정</title>
  <dc:creator>Minhyuk Kim</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-10-20-R_package/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="introduction" class="level1"><h1>Introduction</h1>
<p>이번 포스트에서는 내부 의료연구 프로젝트에서 출발해 <strong>생존분석 모델 성능평가 R 패키지(<code>survC</code>)</strong>를 개발하고, GitHub에 배포하고, CRAN에 제출하기까지의 과정을 공유합니다.</p>
</section><section id="아이디어-선정과-방향-설정---어떤-패키지를-개발할-것인가" class="level1"><h1>아이디어 선정과 방향 설정 - 어떤 패키지를 개발할 것인가?</h1>
<p>패키지 개발의 첫 걸음은 <em>“무엇을 만들 것인가?”</em>를 결정하는 과정입니다. 단순하게 새로운 기능을 추가하는 것이 아닌, 사용자의 불편함을 해결하는 것에서 출발합니다. 내가 ‘평소에 R을 사용하면서 어떤 부분에서 이러한 불편함이 있었지?’ 라는 질문을 스스로에게 묻고 답을 해나가며 개발 아이디어를 얻을 수 있습니다.</p>
<p>고려할 사항들</p>
<ol type="1">
<li>문제 중심의 접근
<ul>
<li>내가 자주 반복하는 과정 중 자동화되면 좋을 부분은 무엇인가?</li>
</ul>
</li>
<li>기존 패키지의 공백 찾기
<ul>
<li>CRAN, GitHub에서 비슷한 기능의 패키지를 찾아보고,</li>
<li>어떤 부분이 불편하게 구현되어 있는가?</li>
<li>어떤 작업은 빠져 있는가?를 비교한다.</li>
</ul>
</li>
<li>최종 사용자가 누구인가?
<ul>
<li>통계 전문가</li>
<li>의료 연구가</li>
<li>비전공 분석가 등</li>
</ul>
</li>
<li>확장성 및 지속 가능성
<ul>
<li>향후 버전에서 확장 가능한 구조인지 고려한다.</li>
</ul>
</li>
</ol>
<p>저는 제가 사용하면서 불편했던 점들을 하나씩 적어보았고, 다른 사람들이 R이나 R Studio를 사용하면서 불편했던 점들도 물어보며 아이디어를 적었습니다.</p>
<p>그 과정에서 나왔던 불편한 점들 중에 몇가지를 적어보았습니다:</p>
<ul>
<li>의료 데이터의 전처리 및 변수 설정을 매번 해야해서 불편함이 있다</li>
<li>의료 데이터에서 배제 포함 조건을 적용한 뒤의 N수를 플로우 차트로 보여주기 어렵다.</li>
<li>의학통계 교육을 진행할 때 필요한 데이터가 여러군데 나뉘어 있어 참조하기 번거롭다.</li>
<li>R studio에서 파일 관리가 번거롭다 등</li>
</ul></section><section id="기획-배경" class="level1"><h1>기획 배경</h1>
<p>의학 연구에서 생존분석 연구를 진행할 때 cox 모델을 만들고 원하는 분석을 진행하려면 여러 패키지를 따로 적용했어야 했으며, 그 과정이 복잡해서 이를 한번에 통합해서 쉽게 결과를 얻을 수 있는 패키지가 있으면 좋겠다고 생각했습니다. 그래서 여러가지 다른 시간대를 기준으로 다른 모델들과의 비교를 위한 <strong>ROC 와 Harrell’s c-index</strong>를 한번에 확인하는 패키지를 만들고 싶었습니다. 그래서 이 결과와 그래프를 보여주는 패키지를 개발하게 되었습니다.</p>
</section><section id="다른-패키지와의-차이점" class="level1"><h1>다른 패키지와의 차이점</h1>
<p><code>survival</code>의 패키지는 생존분석의 모델을 만드는 것에 중점적이지만 <code>survC</code> 패키지는 ROC와 C-index 기반의 모델 평가 산출에 초점이 맞춰져있습니다. <code>riskRegression</code>과 같은 패키지의 경우 모델의 예측 성능을 검증하는데 사용되는데 <code>survC</code> 패키지의 경우 concordance를 중심으로 경량화해서 설계된 패키지라는 것에 차이점이 있습니다. <code>survC</code> 패키지의 장점으로는 원하는 결과값을 한번에 파워포인트 형식으로 추출하고, 이를 파워포인트에서 직접 수정할 수 있다는 점에서 장점이 있습니다.</p>
</section><section id="개발-단계" class="level1"><h1>개발 단계</h1>
<p>각 시간별로 달라지는 생존분석과 이 결과를 나타내는 time-dependent ROC를 구하고 이를 test set과 validation set을 비교했었던 이전에 진행했던 연구를 바탕으로 패키지 개발을 진행하였습니다.</p>
<section id="관련-함수들" class="level2"><h2 class="anchored" data-anchor-id="관련-함수들">관련 함수들</h2>
<p><code>calc_risk_score</code> 함수는 <code>coxph</code> 모델을 기반으로 risk score를 구하는 함수 입니다. <code>cindex_calc</code> 함수는 Harrell’s C‑index을 구할때 사용되는 함수입니다. <code>validation_report</code> 함수는 원하는 결과값을 test와 validate을 비교할 수 있도록 ppt 형식으로 ROC 그래프를 그려주는 함수입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://newjoseph.github.io/survC/">survC</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/therneau/survival">survival</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1234</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/lung.html">lung</a></span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/complete.cases.html">complete.cases</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"time"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"status"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ph.ecog"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Split into training and validation cohorts</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">split_ids</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">train_idx</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">split_ids</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">110</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_idx</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">split_ids</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">111</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">train_df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">train_idx</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_idx</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Fit a simple Cox model on the training data</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cox_fit</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/coxph.html">coxph</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/Surv.html">Surv</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ph.ecog</span>,</span>
<span>  data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">train_df</span>,</span>
<span>  x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Linear predictor / risk scores</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">train_lp</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">calc_risk_score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cox_fit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># risk score 구하기</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_lp</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">calc_risk_score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cox_fit</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># validation dataset risk score 구하기</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  Harrell's concordance on the validation cohort</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cindex_calc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cox_fit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># train dataset의 C-index 구하기</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>Cindex  Lower  Upper 
 0.600  0.534  0.666 </code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">c_index_val</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cindex_calc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cox_fit</span>, newdata <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># validation의 C-index 구하기</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># validation report 추출하기</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">horizons</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">400</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">validation_report</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  train_data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/transform.html">transform</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">train_df</span>, time <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time</span>, status <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/integer.html">as.integer</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  val_data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/transform.html">transform</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">val_df</span>, time <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time</span>, status <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/integer.html">as.integer</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  model <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cox_fit</span>,</span>
<span>  time_col <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"time"</span>,</span>
<span>  status_col <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"status"</span>,</span>
<span>  times <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">horizons</span>,</span>
<span>  time_unit <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"days"</span>,</span>
<span>  output <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"validation_report.pptx"</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
</section><section id="사용한-패키지" class="level2"><h2 class="anchored" data-anchor-id="사용한-패키지">사용한 패키지</h2>
<p>survival, stats, timeROC, officer, rvg 등</p>
</section><section id="패키지-만들기" class="level2"><h2 class="anchored" data-anchor-id="패키지-만들기">패키지 만들기</h2>
<section id="패키지-기본-구조-만들기" class="level3"><h3 class="anchored" data-anchor-id="패키지-기본-구조-만들기">패키지 기본 구조 만들기</h3>
<p><code>devtools</code>로 패키지의 기본 구조를 만듭니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://devtools.r-lib.org/">devtools</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">create_package</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"myPackage"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 원하는 패키지 이름을 정합니다.</span></span></code></pre></div></div>
</div>
<p><code>DESCRIPTION</code>, <code>NAMESPACE</code>, <code>R/</code> 폴더 등을 포함한 패키지의 틀을 자동으로 갖출 수 있게 해줍니다. 그리고 패키지에 필요한 함수들을 <code>R</code> 폴더 아래에 작성합니다.</p>
<p><code>DESCRIPTION</code>은 패키지의 설명을 적은 파일입니다. 어떤 패키지인지, 누가 만들었는지, 다른 패키지는 무엇이 필요한지 명시되어 있습니다.</p>
</section><section id="패키지-설명-추가" class="level3"><h3 class="anchored" data-anchor-id="패키지-설명-추가">패키지 설명 추가</h3>
<p><code>roxygen2</code>를 사용하여 각 함수에 주석을 자동으로 달아줍니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">devtools</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">document</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>위의 함수를 사용해서, <code>NAMESPACE</code>와 <code>man/</code> 폴더에 R Markdown 문서들을 자동으로 생성합니다.</p>
<p><code>NAMESPACE</code>는 패키지의 설계도와 같아서 어떤 함수들이 공개되고, 다른 패키지의 어떤 함수들을 import 할지 적어둔 파일입니다.</p>
<p><code>man/</code> 폴더는 패키지의 각 함수에 대한 설명을 저장하는 폴더입니다.</p>
<p>예시는 아래와 같습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' Compute risk scores from a fitted survival model</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' This helper wraps `stats::predict()` for `coxph` objects so that package users</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' can easily obtain linear predictors (default) or risk scores to feed into</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' downstream metrics such as time-dependent ROC or Harrell's C-index.</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @param model A fitted `coxph` object.</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @param data Optional dataset on which to score the model. Defaults to the</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   training data stored within `model`.</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @param type Scale of the predictions to return. Either `"lp"` (linear</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   predictor, the default) or `"risk"`. If `NULL` or omitted, `"lp"` is used.</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @param ... Additional arguments passed to [stats::predict()].</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @return A numeric vector containing the requested risk scores.</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @export</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @examples</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' if (requireNamespace("survival", quietly = TRUE)) {</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   fit &lt;- survival::coxph(survival::Surv(time, status) ~ age, data = survival::lung)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   # Linear predictor on the training data</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   calc_risk_score(fit)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   # Risk scale predictions on new data</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'   calc_risk_score(fit, survival::lung, type = "risk")</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' }</span></span></code></pre></div></div>
</div>
</section></section><section id="패키지-테스트" class="level2"><h2 class="anchored" data-anchor-id="패키지-테스트">패키지 테스트</h2>
<section id="테스트-코드-작성" class="level3"><h3 class="anchored" data-anchor-id="테스트-코드-작성">테스트 코드 작성</h3>
<p><code>testthat</code> 패키지를 사용하여 unit test를 추가합니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_testthat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_test</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"function_name"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 테스트 하고 싶은 함수 이름을 입력합니다.</span></span></code></pre></div></div>
</div>
<p>위와 같이 프로그램을 실행한다면, <code>tests/testthat/test-function_name.R</code> 파일이 자동으로 생성됩니다. 생성된 파일에 아래와 같이 unit test를 추가할 수 있습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">test_that</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"calc_risk_score matches predict() on training data"</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/therneau/survival">survival</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/lung.html">lung</a></span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/complete.cases.html">complete.cases</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"time"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"status"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/coxph.html">coxph</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/Surv.html">Surv</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lung</span>, x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">expect_equal</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">calc_risk_score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stats</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model</span>, type <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lp"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>기대하는 값과, 실행하는 값의 결과에 따라 테스트의 성공 및 실패 여부를 가릴 수 있습니다. 아래 코드를 실행해서 테스트를 실행할 수 있습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">devtools</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">test</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 테스트 실행</span></span></code></pre></div></div>
</div>
</section></section><section id="패키지-검사" class="level2"><h2 class="anchored" data-anchor-id="패키지-검사">패키지 검사</h2>
<p>위와 같은 과정을 모두 마쳤다면, 전체 패키지를 테스트 합니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">devtools</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">check</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> </span></code></pre></div></div>
</div>
<p>테스트를 진행하면서 ERROR / WARNING / NOTE 메시지를 보고 수정해야합니다. CRAN에 제출하기 위해서는 ERROR, WARNING 없음이 기준입니다.</p>
</section><section id="그-외" class="level2"><h2 class="anchored" data-anchor-id="그-외">그 외</h2>
<section id="패키지-소개문서-작성" class="level3"><h3 class="anchored" data-anchor-id="패키지-소개문서-작성">패키지 소개문서 작성</h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_readme_rmd</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><code>use_readme_rmd()</code> 함수를 사용하여 README 파일을 자동으로 생성하고, 필요에 따라 설명 부분을 직접 작성합니다.</p>
<p><code>pkdown</code>패키지를 사용하여 GitHub의 웹페이지를 만드는 것은 https://blog.zarathu.com/posts/2023-03-17-pkgdown/index.html 를 참조</p>
</section></section></section><section id="오픈소스-운영-및-github-관리" class="level1"><h1>오픈소스 운영 및 GitHub 관리</h1>
<section id="git-생성-및-github-연결" class="level2"><h2 class="anchored" data-anchor-id="git-생성-및-github-연결">Git 생성 및 GitHub 연결</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_git</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># git 생성</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_github</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># github 연결</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">gitcreds</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://gitcreds.r-lib.org/reference/gitcreds_get.html">gitcreds_set</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># GitHub Personal Access Token 입력</span></span></code></pre></div></div>
</div>
<p>패키지의 Git을 생성하고 GitHub 연결합니다.</p>
</section><section id="test-coverage" class="level2"><h2 class="anchored" data-anchor-id="test-coverage">test coverage</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://covr.r-lib.org">covr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="http://covr.r-lib.org/reference/package_coverage.html">package_coverage</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 패키지 폴더에서 실행</span></span></code></pre></div></div>
</div>
<p>test coverage(테스트 커버리지) 는 “테스트 코드가 실제 함수 코드를 얼마나 실행했는가”를 측정하는 지표입니다. 즉, 전체 코드 중 테스트 코드가 다룬 부분의 비율(%)을 보여주는 것입니다. 이 값이 높을수록 테스트가 코드 전체를 잘 검증하고 있다고 볼 수 있습니다.</p>
<pre><code>[![Codecov test coverage](https://codecov.io/gh/USERNAME/REPO/branch/main/graph/badge.svg)](https://app.codecov.io/gh/USERNAME/REPO)</code></pre>
<p>위의 코드를 수정하여 GitHub의 패키지 README 파일에 입력하면, 패키지의 코드 커버리지 결과가 홈페이지에 반영됩니다.</p>
</section><section id="github-자동화" class="level2"><h2 class="anchored" data-anchor-id="github-자동화">GitHub 자동화</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_github_action_check_standard</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># R CMD check 자동화</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_github_action</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pkgdown"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>         <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사이트 자동 빌드</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">usethis</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_github_action</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"test-coverage"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 테스트 커버리지 자동추적</span></span></code></pre></div></div>
</div>
<p>위의 코드를 실행하면, 패키지를 자동으로 체크하고, 패키지 사이트를 자동으로 빌드하며, 테스트 커버리지도 자동으로 추척합니다. 자동화를 설정한 경우에, 업데이트한 경우에 각 단계를 하나씩 확인할 필요 없이, 모든 것을 자동으로 처리할 수 있어 편리합니다.</p>
</section></section><section id="cran에-패키지-제츨하기" class="level1"><h1>CRAN에 패키지 제츨하기</h1>
<section id="패키지-기본-검사" class="level2"><h2 class="anchored" data-anchor-id="패키지-기본-검사">패키지 기본 검사</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb16" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb16-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 패키지 폴더의 상위 디렉토리에서 실행</span></span>
<span id="cb16-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">R</span> CMD build myPackage <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 이름이 myPackage인 패키지를 빌드. tar.gz 파일 생성 </span></span>
<span id="cb16-3"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">R</span> CMD check myPackage_0.1.0.tar.gz <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">--as-cran</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 패키지 버전이 0.1.0을 확인</span></span></code></pre></div></div>
</div>
<p><code>0 errors | 0 warnings | 0 notes</code> 결과가 나와야 CRAN에 제출할 수 있습니다. 주로 점검해야 하는 항목은 아래와 같습니다.</p>
<ul>
<li>예제 코드가 너무 오래 걸리지 않게 작성합니다.</li>
<li>모든 함수에 Rd 문서가 존재해야 합니다.</li>
<li>의존 패키지는 DESCRIPTION의 Imports/Suggests에 명시헤야 합니다.</li>
</ul></section><section id="pdf-메뉴얼-생성-확인" class="level2"><h2 class="anchored" data-anchor-id="pdf-메뉴얼-생성-확인">PDF 메뉴얼 생성 확인</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb17" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb17-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># pdf 메뉴얼 생성 확인</span></span>
<span id="cb17-2"><span class="ex" style="color: null;
background-color: null;
font-style: inherit;">R</span> CMD Rd2pdf .</span></code></pre></div></div>
</div>
<p>pdf 파일의 경우 LaTeX 환경이 설정되어야 합니다. Ubuntu의 경우에는 아래와 같은 코드를 실행하여 LaTeX 환경을 설치할 수 있습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb18" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb18-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sudo</span> apt install texlive texlive-fonts-extra texinfo</span></code></pre></div></div>
</div>
</section><section id="패키지-제출" class="level2"><h2 class="anchored" data-anchor-id="패키지-제출">패키지 제출</h2>
<p>https://cran.r-project.org/submit.html 상기의 링크에 위의 테스트들을 통과한 <code>.tar.gz</code> 파일을 업로드 합니다. 그리고 이메일 인증을 완료하면, CRAN에서 기본적인 테스트를 진행합니다. 오류가 발생한 경우에는 오류 내용과 함께 오류 내용을 수정해서 재제출 하라는 이메일을 10분 내외로 받게 됩니다. 특별한 오류가 발생하지 않은 경우에는, 약 10일 전후로 해서 CRAN의 리뷰 결과를 받게 됩니다.</p>
</section></section><section id="느낀-점" class="level1"><h1>느낀 점</h1>
<p>처음 패키지를 개발하면서 어디서부터 시작해야할지 모르겠다는 막막함이 있었지만, 패키지 개발에 도움을 주는 많은 라이브러리들이 있다는 사실에 놀랐습니다. <code>devtools</code>나 <code>usethis</code>와 같은 라이브러리의 도움을 받아 패키지의 기본 틀을 갖출 수 있었고 이를 토대로 패키지를 개발해서 생각보다 수월했습니다. 그리고 <code>pkgdown</code>와 <code>roxygen2</code>도 사용을 해서 라이브러리의 웹페이지 제작 및 설명을 쉽게 만들수 있었습니다. 패키지의 수정된 부분들을 간단한 함수들을 사용해서 자동으로 업데이트 할 수 있다는 점에서 생각보다 빠른 속도로 패키지를 개발하고 업데이트 할 수 있었습니다.</p>
<p>패키지를 개발하면서 가장 큰 어려움을 겪었던 것은 ‘어떤 패키지를 개발해야 할까?’ 에 대한 답을 찾는 과정이었습니다. digital AI coding assistant를 이용한다면, 원하는 패키지의 개발은 그리 어렵지 않지만, R을 사용하며 불편함을 느꼈던 부분이나, 자동화 혹은 간소화 하고 싶은 부분들은 경험이 많지 않은 사람에게는 쉽지 않겠다는 생각을 했습니다.</p>
<p>처음 패키지를 개발해서 CRAN에 업로드 하는 것보다 앞으로 이 패키지를 유지 보수하고 다른 새로운 기능을 업데이트 하는것이 더 중요하겠다는 생각이 듭니다. 처음에는 미약하지만 R의 생태계를 이해하고 오픈소스의 발전에 기여했다는 점에서 뿌듯함을 느꼈습니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{kim2025,
  author = {Kim, Minhyuk},
  title = {R {패키지} {개발부터} {CRAN} {제출까지의} {여정}},
  date = {2025-10-20},
  url = {https://blog.zarathu.com/posts/2025-10-20-R_package/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-kim2025" class="csl-entry quarto-appendix-citeas">
Kim, Minhyuk. 2025. <span>“R 패키지 개발부터 CRAN 제출까지의
여정.”</span> October 20, 2025. <a href="https://blog.zarathu.com/posts/2025-10-20-R_package/">https://blog.zarathu.com/posts/2025-10-20-R_package/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <category>rpackage</category>
  <category>CRAN</category>
  <category>githubpage</category>
  <guid>https://blog.zarathu.com/posts/2025-10-20-R_package/</guid>
  <pubDate>Mon, 20 Oct 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-10-20-R_package/img/R_logo.png" medium="image" type="image/png" height="144" width="144"/>
</item>
<item>
  <title>반복측정자료 모델링: linear mixed effects model</title>
  <dc:creator>Mingu Jee</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-08-08-lmer and gee/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="개요" class="level1"><h1>개요</h1>
<p>linear mixed model에서 intercept에만 random effects를 주었을 때 추정치들의 standard error가 같아지는 경우에 대해 알아봅시다.</p>
<hr></section><section id="반복측정자료" class="level1"><h1>1. 반복측정자료</h1>
<p>어떤 임상실험에서 연구자가 특정 기간을 주기로 당뇨병 환자들의 인슐린 농도를 측정한다고 해봅시다. 이러한 경우 환자 각각은 서로 다른 사람이므로, <strong>서로 다른 환자에게서 관찰된 데이터</strong>는 서로 <strong>독립</strong>이라고 봐도 무방합니다.</p>
<p>그러나 <strong>동일한 환자에게서 반복적으로 측정된 값</strong>들은 서로 독립이라고 보기 어렵습니다. 동일한 환자에게서 측정정되었기 때문에, 각 시점의 데이터들 간에는 특정한 <strong>상관관계가 존재</strong>할 것이라는 생각을 해볼 수 있겠습니다. 예컨대, 당뇨임에도 치료 효과가 좋은 환자는 시간이 지날수록 인슐린 농도가 점점 높아질 수 있고, 치료에 소극적인 환자의 경우에는 반대의 경향성을 보일 것입니다.</p>
<p>일반적으로 회귀분석에서는 모든 관측지(y, 종속변수)들이 서로 <strong>독립</strong>이라고 가정합니다. 그러나 반복측정자료의 경우 위와 같은 이유 때문에 회귀분석을 통해 분석하기에 어려움이 존재합니다. 이에 따라 반복측정자료를 분석하는 통계적인 방법론으로는 <strong>Mixed Effect Model</strong>을 사용하는 방법과, <strong>Generalized Estimating Equations</strong>을 이용하는 방법 등이 존재합니다. 우선 Mixed Effect Model 중 <strong>Linear Mixed Effect Model</strong>에 대해 알아보도록 하겠습니다.</p>
</section><section id="fixed-random-effects" class="level1"><h1>2. Fixed &amp; Random Effects</h1>
<p>모델에 대해 본격적으로 살펴보기 전에, <strong>fixed effects</strong>와 <strong>random effects</strong>에 대해 간단하게 알아보겠습니다.</p>
<section id="fixed-effects" class="level2"><h2 class="anchored" data-anchor-id="fixed-effects">Fixed Effects</h2>
<p>우선 <strong>fixed effects</strong>의 경우, 우리가 관심 있는 <strong>전체 모집단의 평균적인 효과를 추정하는 모수</strong>입니다. 예를 들어, 다음과 같은 모델에 대해 생각해보겠습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?y_%7Bij%7D%20=%20%5Cbeta_0%20+%20%5Cbeta_1%20*%20Group_i%20+%20%5Cbeta_2%20*%20Time_%7Bij%7D%20+%20%5Cbeta_3(Group_i%20*%20Time_%7Bij%7D)%20+%20%5Cepsilon_%7Bij%7D"></p>
<p><img src="https://latex.codecogs.com/png.latex?y_%7Bij%7D">는 i번째 환자의 j번째 측정값을 의미하고, <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_%7Bij%7D%5Csim%20N(0,%20%5Csigma%5E2)">라고 가정하겠습니다.</p>
<p>위 모델에서</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cbeta_1">(<strong>Group effect</strong>): 시간에 관계없이 Group간 평균적인 차이를 나타냅니다. 이는 baseline에서 그룹 간 평균 차이를 의미합니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cbeta_2">(<strong>Time effect</strong>): 어떤 Group에 속하는지와 무관하게 시간이 지남에 따라 전체 평균이 얼마나 변하는지를 나타냅니다. 이 값이 음수라면 평균이 감소, 양수면 평균이 증가하는 경향성을 의미합니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cbeta_3">(<strong>Interaction effect</strong>): 시간과 Group 간의 상호작용 효과로, 시간에 따라 Group별로 나타나는 차이를 나타냅니다. 즉, 시간에 따라 Group 별로 얼마나 평균이 다르게 나타났는지를 의미합니다.</p>
<p>fixed effects의 경우 위와 같이 estimate 형태로 도출되고, 전체 데이터에 일관되게 적용됩니다.</p>
</section><section id="random-effects" class="level2"><h2 class="anchored" data-anchor-id="random-effects">Random Effects</h2>
<p><strong>random effects</strong>의 경우 fixed effects와 비슷하면서도 조금 다른 형태를 보입니다. 이번 포스팅에서는 <strong>intercept</strong>에만 random effect를 주어 여기에 집중해보도록 하겠습니다.</p>
<p>random effects는 이름 그대로 무작위적인 부분 즉, <strong>확률적인 효과</strong>를 모델링 합니다. 앞서 언급했듯 fixed effects가 우리가 관심 있는 모집단 전체의 평균적인 효과를 추정하는 모수였다면, random effects는 그 <strong>모집단에서 추출된 개별 sample이나 cluster가 갖는 특징이나 변동성</strong>을 잡아내는 <strong>확률변수(random variable)</strong>입니다.</p>
<p>앞서 언급한 당뇨병 환자들을 대상으로 한 임상실험을 다시 생각해봅시다. 모든 환자들이 동일한 치료를 받더라도, 치료 효과가 나타나는 시작점이나(baseline)이나 시간에 따른 변화 정도는 환자 개개인마다 다를 수 있습니다. 이러한 <strong>이질성(heterogeneity)</strong>을 설명해주는 것이 <strong>random effects</strong>입니다. 즉, 각 subject가 전체적인 평균(<img src="https://latex.codecogs.com/png.latex?%5Cbeta_0">)으로부터 얼마나 벗어나 있는지를 나타내는 <strong>확률변수</strong>로 활용됩니다. 아래 모델을 보겠습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?y_%7Bij%7D%20=%20(u_%7Bi%7D%20+%5Cbeta_0)%20+%20%5Cbeta_1%20*%20Group_i%20+%20%5Cbeta_2%20*%20Time_%7Bij%7D%20+%20%5Cbeta_3(Group_i%20*%20Time_%7Bij%7D)%20+%20%5Cepsilon_%7Bij%7D"></p>
<p>여기서 새로 추가된 <img src="https://latex.codecogs.com/png.latex?u_i">가 바로 i번째 환자의 random effect입니다. 이는 특정 상수가 아니라 <img src="https://latex.codecogs.com/png.latex?u_i%20%5Csim%20N(0,%20%5Csigma_u%5E2)">을 따르는 확률변수이며, 모든 <img src="https://latex.codecogs.com/png.latex?i,%20j">에 대해 <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_%7Bij%7D">과 독립이라고 가정합니다.</p>
<p>위 모델은 결국 다음을 가정합니다.</p>
<ol type="1">
<li><p>모든 환자들은 평균적인 시작점을 공유하지만, 각 환자는 자신만의 고유한 시작점(intercept)을 (<img src="https://latex.codecogs.com/png.latex?u_i%20+%5Cbeta_0">) 갖습니다(<strong>random effects</strong>).</p></li>
<li><p>Group, Time, Interaction effects는 모든 환자에게 동일하하게 적용됩니다(<strong>fixed effects</strong>).</p></li>
</ol>
<p>이처럼 모델에 random effects를 포함하게 되면, 단순하게 모든 환자의 평균적인 경향성만을 보는 것에서 한 발 더 나아가, 환자들 사이에 존재하는 변동까지 고려하게 되어 데이터를 훨씬 더 정교하게 설명할 수 있습니다.</p>
<p>결국 random effects가 표현하고자 하는 것은 <strong>‘변동’</strong>에 초점을 맞추기에, fixed effects는 <strong>기댓값</strong>에 해당하는 <img src="https://latex.codecogs.com/png.latex?E%5By_%7Bij%7D%5D">를 모델링 한다면, random effects는 <strong>분산(공분산)</strong>에 해당하는 <img src="https://latex.codecogs.com/png.latex?Var(y_%7Bij%7D)">를 모델링 합니다.</p>
<p>실제로 이 모델을 통해 i번째 환자의 첫번째와 두번째 시점의 공분산을 구해보면, <img src="https://latex.codecogs.com/png.latex?Cov(y_%7Bi1%7D,y_%7Bi2%7D)%20=%20Cov(u_i+%5Cepsilon_%7Bi1%7D,%20u_i+%5Cepsilon_%7Bi2%7D)"> <img src="https://latex.codecogs.com/png.latex?=%20Cov(u_i,u_i)%20+%20Cov(u_i,%5Cepsilon_%7Bi1%7D)%20+%20Cov(u_i,%5Cepsilon_%7Bi2%7D)%20+%20Cov(%5Cepsilon_%7Bi1%7D,%20%5Cepsilon_%7Bi2%7D)"> <img src="https://latex.codecogs.com/png.latex?=%20%5Csigma_u%5E2">으로 도출되므로, 같은 환자에게서 다른 시간대에 측정된 정보는 상관되어 있음을 확인할 수 있습니다.</p>
<hr></section></section><section id="linear-mixed-effect-model-with-random-intercept" class="level1"><h1>3. Linear Mixed Effect Model with Random Intercept</h1>
<p>자 이제 intercept에만 random effects가 주어진 상태에서 어떤 경우에 standard error가 같아지는지 알아보겠습니다. 이를 만족하기 위해서는 다음과 같은 엄격한 조건이 모두 성립해야합니다.</p>
<ol type="1">
<li><p><strong>Complete Case</strong>: 주어진 데이터에서 <strong>결측이 단 하나라도 존재해서는 안 됩니다</strong>.</p></li>
<li><p><strong>Balanced Data</strong>: <strong>연구 디자인이 완벽하게 대칭적이고 균형 잡혀있어야</strong> 합니다. 반복측정자료에서 <strong>균형</strong>의 의미는 크게 두 가지를 의미합니다. 첫째는 <strong>관측 횟수의 균형</strong>으로 <strong>모든 환자가 동일한 횟수로 측정</strong>되어야하며, 둘째는 <strong>관측 시점의 균형</strong>으로, <strong>모든 환자가 동일한 시점에 측정</strong>되어야 합니다.</p></li>
<li><p><strong>Compound Symmtery</strong>: <strong>모든 반복측정자료들 사이의 상관관계가 동일</strong>해야 합니다. 즉, 모든 측정 시점에서 각 데이터의 분산은 동일해야 하고<img src="https://latex.codecogs.com/png.latex?(%5Cforall%20i,%20j,%5C%20%20Var(Y_%7Bij%7D)%20=%20%5Csigma%5E2%20+%20%7B%5Csigma_u%7D%5E2)">, 각 환자 내에서 임의의 두 시점간 공분산 또한 항상 동일해야 합니다<img src="https://latex.codecogs.com/png.latex?(Cov(Y_%7Bij%7D,%20Y_%7Bik%7D)=%20%7B%5Csigma_u%7D%5E2%20%5Ctext%7B,%20where%20%7D%20j%5Cneq%20k)">.</p></li>
</ol>
<p>예시 데이터를 확인해보겠습니다. 참고로 예시 데이터의 값들은 모두 임의로 채워넣었기에 결과가 유의하고 않고는 의미가 없음을 미리 말씀드립니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://ycphs.github.io/openxlsx/index.html">openxlsx</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">openxlsx</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">read.xlsx</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"data.xlsx"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/dim.html">dim</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>[1] 80  4</code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>   NO          Time Group     Value
1   1        Pre OP 65-79 14.786093
2   1  Post 1 month 65-79 10.477890
3   1 Post 6 months 65-79 10.196241
4   1   Post 1 year 65-79  7.049579
5   2        Pre OP 65-79  9.057036
6   2  Post 1 month 65-79 10.267466
7   2 Post 6 months 65-79  9.657085
8   2   Post 1 year 65-79 10.387103
9   3        Pre OP 65-79  8.508681
10  3  Post 1 month 65-79 10.999663</code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">tail</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>   NO          Time Group     Value
71 18 Post 6 months   ≥80  6.702226
72 18   Post 1 year   ≥80 10.030724
73 19        Pre OP   ≥80  9.320332
74 19  Post 1 month   ≥80  9.939527
75 19 Post 6 months   ≥80 10.699433
76 19   Post 1 year   ≥80 10.021359
77 20        Pre OP   ≥80  8.859701
78 20  Post 1 month   ≥80  9.440627
79 20 Post 6 months   ≥80 10.545564
80 20   Post 1 year   ≥80  8.391252</code></pre>
</div>
</div>
<p>이 데이터는 총 20명의 환자를 각 4번씩 반복측정한 데이터입니다. NO열은 각 환자의 ID를 의미하고, Time은 총 4개로 수술 직후, 1개월 후, 6개월 후, 1년 후로 나뉘어 있습니다. Group의 경우 65~79세인 경우(NO가 1-10인 환자)와 80세 이상(NO가 11-20인 환자)인 두 가지 경우가 존재합니다. 이 데이터는 결측이 하나도 존재하지 않고 20명의 환자에 대해 각각 모두 같은 시점에 같은 횟수가 측정된 <strong>Complete &amp; Balanced data</strong>입니다. 우리는 이러한 반복측정자료에서 Time과 Group에 대한 정보로 Value값에 대해 예측하고 싶어하는 상황입니다. Compounds Symmetry만 만족한다면, 앞서 말했던 것처럼 모든 S.E. 값이 동일하게 나올 것입니다. 한 번 확인해보겠습니다.</p>
<p>lme4패키지의 lmer 함수를 사용하여 진행해보면 아래와 같은 결과를 얻을 수 있습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/lme4/lme4/">lme4</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stderr">
<pre><code>Loading required package: Matrix</code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lmer_fit</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lmer</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Value</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Time</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">NO</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lmer_fit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>Linear mixed model fit by REML ['lmerMod']
Formula: Value ~ Group * Time + (1 | NO)
   Data: data

REML criterion at convergence: 310.5

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.2756 -0.4501  0.0719  0.4712  2.6606 

Random effects:
 Groups   Name        Variance Std.Dev.
 NO       (Intercept) 0.04915  0.2217  
 Residual             3.33376  1.8259  
Number of obs: 80, groups:  NO, 20

Fixed effects:
                             Estimate Std. Error t value
(Intercept)                    9.9769     0.5816  17.153
Group65-79                     0.2331     0.8225   0.283
TimePost 1 year               -0.4466     0.8165  -0.547
TimePost 6 months              0.4467     0.8165   0.547
TimePre OP                    -0.9778     0.8165  -1.197
Group65-79:TimePost 1 year     0.2448     1.1548   0.212
Group65-79:TimePost 6 months  -0.1802     1.1548  -0.156
Group65-79:TimePre OP          0.7250     1.1548   0.628

Correlation of Fixed Effects:
            (Intr) G65-79 TmPs1y TmPs6m TmPrOP G65-1y G65-6m
Group65-79  -0.707                                          
TimePost1yr -0.702  0.496                                   
TmPst6mnths -0.702  0.496  0.500                            
TimePre OP  -0.702  0.496  0.500  0.500                     
G65-79:TP1y  0.496 -0.702 -0.707 -0.354 -0.354              
G65-79:TP6m  0.496 -0.702 -0.354 -0.707 -0.354  0.500       
G65-79:TPOP  0.496 -0.702 -0.354 -0.354 -0.707  0.500  0.500</code></pre>
</div>
</div>
<p>summary의 Random effects를 보면 intercept에 대해서만 적용된 모습을 확인할 수 있습니다. Fixed effects의 Std.Error을 보게 되면 time effect와 interaction effect에 해당하는 부분의 <strong>Std. Error가 모두 같은 모습</strong>을 확인할 수 있습니다. 어째서 이런 결과가 나타나는 것일까요?</p>
<section id="linear-regression-model" class="level2"><h2 class="anchored" data-anchor-id="linear-regression-model">Linear Regression Model</h2>
<p>일반적인 선형 회귀 모델을 행렬 형태로 표현하면 다음과 같이 나타낼 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?Y%20=%20X%5Cbeta%20+%20%5Cepsilon"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Cepsilon%20%5Coverset%7B%5Cmathrm%7Biid%7D%7D%7B%5Csim%7D%20N(0,%20%5Csigma%5E2I)">이라고 가정하는 것이 일반적입니다. 이후 최소제곱법(LSE)를 사용하여 <img src="https://latex.codecogs.com/png.latex?%5Chat%5Cbeta=(X%5ETX)%5E%7B-1%7DX%5ETY">라는 추정량을 얻을 수 있고, <img src="https://latex.codecogs.com/png.latex?Var(%5Chat%5Cbeta)=%5Csigma%5E2(X%5ETX%5E%7B-1%7D)">라고 계산할 수 있습니다.</p>
</section><section id="linear-mixed-model" class="level2"><h2 class="anchored" data-anchor-id="linear-mixed-model">Linear Mixed Model</h2>
<p>지금 우리가 다루고 있는 모델은 여기에 상관관계를 위해 <img src="https://latex.codecogs.com/png.latex?u">라는 term을 추가한 linear mixed model이고, 오차의 독립성 가정을 하지 않기에 선형 회귀 모델과는 조금 다른 구조와 분산-공분산 행렬을 가집니다. 이를 아래와 같이 표현할 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?Y%20=%20X%5Cbeta+Zu+%20%5Cepsilon"></p>
<p><img src="https://latex.codecogs.com/png.latex?u%20%5Coverset%7B%5Cmathrm%7Biid%7D%7D%7B%5Csim%7D%20N(0,%20%5Csigma_u%5E2I),%20%5C%20%5Cepsilon%20%5Coverset%7B%5Cmathrm%7Biid%7D%7D%7B%5Csim%7D%20N(0,%20%5Csigma%5E2I)">이고, 이 둘은 독립이라고 가정하겠습니다.</p>
<p>i번째 환자 데이터의 분산-공분산 구조를 살펴보면 다음과 같습니다. <img src="https://latex.codecogs.com/png.latex?%0AVar(Y_i)%20=%20V_i%20=%20Var(X_i%5Cbeta+Z_iu_i+%5Cepsilon_i)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20Var(Z_iu_i+%5Cepsilon_i)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20Var(Z_iu_i)%20+%20Var(%5Cepsilon_i)%20(%5Cbecause%20%5Cepsilon%20%5Cperp%5C!%5C!%5C!%5Cperp%20u)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20Z_iVar(u_i)Z_i%5ET%20+%20Var(%5Cepsilon_i)%0A"></p>
<p>총 20명의 환자가 4번씩 반복측정한 상황이고, random effects가 intercept term에만 존재하기에 <img src="https://latex.codecogs.com/png.latex?u_i">는 univariate normal distribution을 따르는 확률변수가 되고, <img src="https://latex.codecogs.com/png.latex?Z_i%20=%20%5Cbegin%7Bbmatrix%7D%0A1%20&amp;%201%20&amp;%201%20&amp;%201%0A%5Cend%7Bbmatrix%7D%20"> 라고 할 수 있으므로 <img src="https://latex.codecogs.com/png.latex?%0AV_i%20=%20%5Cbegin%7Bbmatrix%7D%0A1%20&amp;%201%20&amp;%201%20&amp;%201%0A%5Cend%7Bbmatrix%7D%0A%5Csigma_u%5E2%0A%5Cbegin%7Bbmatrix%7D%0A1%20%5C%5C%201%20%5C%5C%201%20%5C%5C%201%0A%5Cend%7Bbmatrix%7D%20+%0A%5Csigma%5E2%0A%5Cbegin%7Bbmatrix%7D%0A1%20&amp;%200%20&amp;%200%20&amp;%200%20%5C%5C%0A0%20&amp;%201%20&amp;%200%20&amp;%200%20%5C%5C%0A0%20&amp;%200%20&amp;%201%20&amp;%200%20%5C%5C%0A0%20&amp;%200%20&amp;%200%20&amp;%201%0A%5Cend%7Bbmatrix%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cbegin%7Bbmatrix%7D%0A%5Csigma%5E2+%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20%5C%5C%0A%5Csigma_u%5E2%20&amp;%20%5Csigma%5E2+%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20%5C%5C%0A%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20&amp;%20%5Csigma%5E2+%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20%5C%5C%0A%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20&amp;%20%5Csigma_u%5E2%20&amp;%20%5Csigma%5E2+%5Csigma_u%5E2%0A%5Cend%7Bbmatrix%7D%0A"> 라는 결과를 얻게 됩니다. 이를 통해 random intercept 모델은 <strong>compound symmetry</strong> 구조를 만족함을 확인할 수 있습니다.</p>
<p>자 그럼 이제 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">의 추정치를 구해보겠습니다. 일반적인 Linear Regression Model에서 OLS(Ordinary Least Square)방법을 이용해 <img src="https://latex.codecogs.com/png.latex?%5Chat%5Cbeta=(X%5ETX)%5E%7B-1%7DX%5ETY">를 도출할 수 있다고 했고, 대략적인 과정은 다음과 같습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY%20=%20X%5Cbeta%20+%20%5Cepsilon%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BLet%20%7D%0AQ(%5Cbeta)%20=%20(Y%20-%20X%5Cbeta)%5ET(Y-X%5Cbeta)%0A=%20Y%5ETY%20-2Y%5ETX%5Cbeta%20+%5Cbeta%20X%5ETX%20%5Cbeta%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BThen%20%7D%20%5Carg%5Cmin_%5Cbeta%20Q(%5Cbeta)=%5Chat%5Cbeta%20%5Ctext%7B%20would%20be%20the%20vector%20that%20satisfies%20%7D%20%5Cfrac%7B%5Cpartial%20Q(%5Cbeta)%7D%7B%5Cpartial%20%5Cbeta%7D%20=%200%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7Bsince%20%7DQ(%5Cbeta)%20%5Ctext%7B%20is%20a%20convex%20function%20w.r.t%20%7D%20%5Cbeta.%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BWe%20know%20that%20%7DX%5ETX%20%5Ctext%7B%20is%20symmetric%20and%20by%20assuming%20it's%20invertible,%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20Q(%5Cbeta)%7D%7B%5Cpartial%20%5Cbeta%7D%20=%20-2X%5ETY%20+%202X%5ETX%5Cbeta%20=%200%0A"> <img src="https://latex.codecogs.com/png.latex?%0AX%5ETX%5Cbeta%20=%20X%5ETY%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctherefore%20%5Chat%5Cbeta%20=%20(X%5ETX)%5E%7B-1%7DX%5ETY%0A"> 자 이제는 이 방법을 살짝 비틀어서 Linear Mixed Model의 경우를 살펴보겠습니다. 일반적인 형태의 모델은 다음과 같았습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY%20=%20X%5Cbeta+Zu+%20%5Cepsilon%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BLet%20%7D%20Zu+%5Cepsilon%20=%20%5Cepsilon%5E*%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BThen%20%7DY%20=%20X%5Cbeta%20+%20%5Cepsilon%5E*%0A"> <img src="https://latex.codecogs.com/png.latex?Var(Y)%20=%20V">는 <strong>symmertic &amp; positive-semi-definite matrix</strong>이므로 <strong>Cholesky Decomposition</strong>에 의해 다음을 만족하는 가역인 행렬 <img src="https://latex.codecogs.com/png.latex?%5CSigma">가 존재합니다. <img src="https://latex.codecogs.com/png.latex?V%20=%20%5CSigma%20%5CSigma%5ET">. 이제 <img src="https://latex.codecogs.com/png.latex?%5CSigma%5E%7B-1%7D">을 위 모델의 양변에 곱해봅시다. <img src="https://latex.codecogs.com/png.latex?%0A%5CSigma%5E%7B-1%7D%20Y%20=%20%5CSigma%5E%7B-1%7DX%5Cbeta%20+%20%5CSigma%5E%7B-1%7D%5Cepsilon%5E*%0A"> 이제 아래와 같이 변수들을 새로 정의하면, <img src="https://latex.codecogs.com/png.latex?%0A%5CSigma%5E%7B-1%7D%20Y%20=%20%5Ctilde%20Y,%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5CSigma%5E%7B-1%7DX%5Cbeta%20=%20%5Ctilde%20X%5Cbeta%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CSigma%5E%7B-1%7D%5Cepsilon%5E*%20=%20%5Ctilde%20%5Cepsilon%0A"></p>
<p>아래와 같이 OLS의 경우와 아주 유사한 형태의 모델을 새로 정의할 수 있습니다. <img src="https://latex.codecogs.com/png.latex?%0A%5Ctilde%20Y%20=%20%5Ctilde%20X%5Cbeta%20+%20%5Ctilde%20%5Cepsilon%0A"> 이러한 방법을 OLS를 일반화했다고 하여 <strong>Generalized Least Square(GLS)</strong>라고 부릅니다. 이제 OLS와 형태가 같기 때문에 <img src="https://latex.codecogs.com/png.latex?%5Chat%5Cbeta">의 추정량은 다음와 같이 도출됩니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%5Cbeta_%7BGLS%7D%20=%20(%5Ctilde%20X%5ET%5Ctilde%20X)%5E%7B-1%7D%5Ctilde%20X%5ET%5Ctilde%20Y%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20((%5CSigma%5E%7B-1%7DX)%5ET%5CSigma%5E%7B-1%7DX)%5E%7B-1%7D(%5CSigma%5E%7B-1%7DX)%5ET%20%5CSigma%5E%7B-1%7DY%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ET%7B%5CSigma%5ET%7D%5E%7B-1%7D%5CSigma%5E%7B-1%7DX)%5E%7B-1%7DX%5ET%7B%5CSigma%5ET%7D%5E%7B-1%7D%5CSigma%5E%7B-1%7DY%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ET(%5CSigma%5ET%5CSigma)%5E%7B-1%7DX)%5E%7B-1%7DX%5ET(%5CSigma%5ET%5CSigma)%5E%7B-1%7DY%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ET(%5CSigma%5CSigma%5ET)%5E%7B-1%7DX)%5E%7B-1%7DX%5ET(%5CSigma%5CSigma%5ET)%5E%7B-1%7DY%20%5C%20(%5Cbecause%20%5CSigma%5ET%20=%5CSigma)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7DY%0A"> 임을 알 수 있습니다. 그렇다면 분산-공분산 행렬은 다음과 같이 유도될 것입니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AVar(%5Chat%5Cbeta_%7BGLS%7D)%20=%20Var((X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7DY)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D%20Var(Y)%20((X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D)%5ET%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7DV%20%7BV%5E%7B-1%7D%7D%5ETX%20%7B(X%5ETV%5E%7B-1%7DX)%5E%7BT%7D%7D%5E%7B-1%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D(X%5ETV%5E%7B-1%7DX)%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D%20%5C%20(%5Cbecause%20VV%5E%7B-1%7D=%20I,%20%7BV%5E%7B-1%7D%7D%5ET=V%5ET)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D%0A"> 앞에서 보인것 처럼 <img src="https://latex.codecogs.com/png.latex?V">는 <strong>Compound Symmertry</strong> 구조이고, <strong>Complete &amp; Balanced</strong>한 데이터를 다루고 있기에 Design matrix인 X의 구조도 변하지 않습니다. 결국 standard error을 결정짓는 것은 방금 전 구한 <img src="https://latex.codecogs.com/png.latex?Var(%5Chat%5Cbeta_%7BGLS%7D)">의 대각성분과 관련이 있을텐데, 이 값이 같기 때문에 fixed effects의 standard error가 같게 나오는 것입니다.</p>
</section><section id="why-linear-mixed-model-is-better" class="level2"><h2 class="anchored" data-anchor-id="why-linear-mixed-model-is-better">Why Linear Mixed Model Is Better?</h2>
<p>지금까지 잘 읽어오시다가 몇몇 분들은 이런 생각이 드셨을 수도 있습니다.</p>
<p><em>반복측정자료고 뭐고 그냥 환자 NO를 dummy variable로 만들어서 fitting해도 되지 않나? 굳이 왜 random effects를 쓰지? 이렇게 하면 fixed effects만으로도 표현이 가능한데…</em></p>
<p>네 맞습니다. 가능합니다. 한 번 그렇게 진행해볼까요?</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">glm_fit</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Value</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">NO</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Time</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Time</span>,</span>
<span>  data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">glm_fit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>
Call:
glm(formula = Value ~ NO + Group + Time + Group * Time, data = data)

Coefficients: (1 not defined because of singularities)
                             Estimate Std. Error t value Pr(&gt;|t|)    
(Intercept)                  10.67448    1.04090  10.255 2.78e-14 ***
NO10                          0.43968    1.29108   0.341   0.7348    
NO11                         -0.24069    1.47206  -0.164   0.8707    
NO12                         -0.88679    1.47206  -0.602   0.5494    
NO13                          0.48099    1.47206   0.327   0.7451    
NO14                         -0.71105    1.47206  -0.483   0.6310    
NO15                          0.28560    1.47206   0.194   0.8469    
NO16                         -2.18372    1.47206  -1.483   0.1438    
NO17                          0.75654    1.47206   0.514   0.6094    
NO18                         -2.92089    1.47206  -1.984   0.0523 .  
NO19                         -0.43488    1.47206  -0.295   0.7688    
NO2                          -0.78528    1.29108  -0.608   0.5456    
NO20                         -1.12076    1.47206  -0.761   0.4498    
NO3                          -0.50924    1.29108  -0.394   0.6948    
NO4                          -1.20306    1.29108  -0.932   0.3556    
NO5                          -0.20593    1.29108  -0.160   0.8739    
NO6                          -1.05182    1.29108  -0.815   0.4188    
NO7                          -0.05504    1.29108  -0.043   0.9662    
NO8                          -1.48829    1.29108  -1.153   0.2541    
NO9                           0.21424    1.29108   0.166   0.8688    
Group65-79                         NA         NA      NA       NA    
TimePost 1 year              -0.44664    0.81655  -0.547   0.5866    
TimePost 6 months             0.44668    0.81655   0.547   0.5866    
TimePre OP                   -0.97777    0.81655  -1.197   0.2364    
Group65-79:TimePost 1 year    0.24482    1.15478   0.212   0.8329    
Group65-79:TimePost 6 months -0.18020    1.15478  -0.156   0.8766    
Group65-79:TimePre OP         0.72501    1.15478   0.628   0.5328    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for gaussian family taken to be 3.333764)

    Null deviance: 260.10  on 79  degrees of freedom
Residual deviance: 180.02  on 54  degrees of freedom
AIC: 345.91

Number of Fisher Scoring iterations: 2</code></pre>
</div>
</div>
<p>Group 변수에서 <strong>NA</strong>가 나왔습니다. 아마 다중공산성 문제 등이 발생해서 design matirx가 singular 해진 것 같습니다. 여기서부터 문제가 느껴지시나요? NO의 level이 너무 많아지게되어서 <strong>모델 구조가 필요 이상으로 복잡</strong>해지고, 원하지도 않은 수많은 계수들을 추정하게 됩니다. 이렇게 되면 정작 우리가 중요하게 생각하고 있는 변수들의 효과에 대한 <strong>해석도 어려워집니다.</strong></p>
<p>Group 변수에서 NA가 발생한 것 자체가 <strong>추정의 불안정성</strong>을 보여주는 예시라고 볼 수 있겠습니다. 이와 별개로 만약 반복측정된 횟수가 정말 적은 경우라면, NA가 뜨지 않더라고 해당 변수에 해당하는 표준 오차가 굉장히 커질 것이기에 값 자체에 대한 신뢰성 또한 떨어집니다.</p>
<p>이것이 우리가 lmer을 쓰는 이유 중 하나입니다. fixed effects만으로 처리하기에는 어려운 부분을 random effects로 말끔하게 처리하여 문제를 해결해준다는 점에서 큰 장점을 가집니다.</p>
</section></section><section id="gee-vs-lmer" class="level1"><h1>4. GEE VS lmer</h1>
<section id="gee" class="level2"><h2 class="anchored" data-anchor-id="gee">GEE</h2>
<p>이번에는 GEE(Generaluzed Estimating Equation)를 사용해 똑같은 데이터를 분석해보겠습니다. 코드는 아래와 같습니다. 참고로 <strong>corsrt = exchangeable</strong>이라고 설정해주어 한 subject내에서 측정된 모든 관측지의 상관관계는 동일하다고 가정했습니다. GEE는 심지어 반복측정환자들끼리 잘 정리되어 있는 데이터가 아니라면 제대로 돌아가지 않습니다. 예시로 사용한 데이터는 반복측정된 환자 순서대로 잘 정리되어 있음을 밝힙니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">geepack</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">gee_mod</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geeglm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Value</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Time</span>,</span>
<span>  id        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">NO</span>,</span>
<span>  data      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span>,</span>
<span>  corstr    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"exchangeable"</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">gee_mod</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>
Call:
geeglm(formula = Value ~ Group * Time, data = data, id = NO, 
    corstr = "exchangeable")

 Coefficients:
                             Estimate Std.err    Wald Pr(&gt;|W|)    
(Intercept)                    9.9769  0.8306 144.290   &lt;2e-16 ***
Group65-79                     0.2331  0.9386   0.062    0.804    
TimePost 1 year               -0.4466  0.9007   0.246    0.620    
TimePost 6 months              0.4467  0.4787   0.871    0.351    
TimePre OP                    -0.9778  0.8964   1.190    0.275    
Group65-79:TimePost 1 year     0.2448  1.2324   0.039    0.843    
Group65-79:TimePost 6 months  -0.1802  0.6812   0.070    0.791    
Group65-79:TimePre OP          0.7250  1.3258   0.299    0.584    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Correlation structure = exchangeable 
Estimated Scale Parameters:

            Estimate Std.err
(Intercept)    3.045  0.7025
  Link = identity 

Estimated Correlation Parameters:
      Estimate Std.err
alpha  0.01453 0.07475
Number of clusters:   20  Maximum cluster size: 4 </code></pre>
</div>
</div>
<p>lmer을 사용했을 때와 다른 점을 느끼셨나요? 두 모델 모두 <strong>estimate들의 값은 같게 나오지만, standard error의 값이 다르게 나온다</strong>는 점입니다. 두 방법 모두 반복측정자료를 분석하는데 사용되는데, 대체 어떤 차이가 있는 것일까요?</p>
<p>저는 <strong>To GEE or Not to GEE: Comparing Population Average and Mixed Models for Estimating the Associations Between Neighborhood Risk Factors and Health</strong>라는 논문에서 어느 정도 해답을 찾을 수 있었습니다.</p>
<p><strong>lmer</strong>과 같은 Mixed Model을 <strong>neighbor-specific model</strong>로 설명합니다. 이름에서 알 수 있듯이, 특정한 지역(neighbor or cluster) 혹은 객체(subject)가 각자 <strong>자신만의 고유한 반응 패턴</strong>을 가질 수 있다고 가정합니다. 결국 앞서 계속 살펴봤던 random effects model과 같은 말입니다. 각 지역(혹은 개인)마다 다른 intercept나 slope를 가질 수 있다고 가정하여 개별 단위 수준에서 추정을 진행합니다. 각 환자마다 당뇨병 치료의 영향이 다를 수 있음을 인정하는 것입니다.</p>
<p>반면, <strong>GEE</strong>와 같은 모델들은 <strong>Population-Average Model</strong>로 설명합니다. 이 경우, 각 지역이나 객체의 차이보다는 <strong>전체 모집단에서 나타나는 평균적인 관계</strong>에 더 주목합니다. 당뇨병 치료에서 어떤 요인이 한 단위 증가할 때, 어떤 환자인지는 중요하지 않고, 평균적으로 얼마나 인슐린 농도가 변하는지를 탐구하고자 하는 것입니다.</p>
<p>논문에 따르면, linear model에서 상당히 흥미로운 부분을 찾을 수 있습니다.</p>
<p><em>A simple random intercept linear model implies equal variances for all observations and equal covariances of all possible paired observations within the statistical unit (neighborhood) … This will yield the same estimates as the exchangeable working correlation model in GEE.</em></p>
<p>random intercept만을 포함하는 lmer 모델은 compound symmertry를 만족하기에 모든 관측치의 분산이 같고, 한 subject내의 모든 관측치는 동일한 상관관계를 공유합니다. 나아가 이는 GEE에서 상관구조를 ’exchangeable’로 설정했을 때의 가정과 <strong>정확히 일치</strong>합니다. 이러한 이유에서 이 둘은 동일한 회귀 계수 추정치를 내놓았던 것입니다.</p>
<p>그러나 통계적인 추론 과정에서 사용되는 standard error는 달라집니다. lmer의 경우 앞서 <img src="https://latex.codecogs.com/png.latex?u%5Csim%20N(0,%20%5Csigma_u%5E2)">라고 가정한 것처럼, <strong>random effects의 분포</strong>를 가정합니다. 당연히 이 가정이 맞다는 전제에서 <strong>MLE(Maximum Likelihood Estimation)</strong>나 <strong>REML(Restrictied Maximum Likelihood)</strong>을 이용해 standard error를 계산합니다. 모델의 구조적 가정이 그대로 반영되기 때문에, 모든 시점에서 동일한 표준 오차를 내놓을 수밖에 없는 것입나다.</p>
<p>GEE의 경우 lmer과 달리 <strong>random effects의 확률적인 분포를 가정하지 않습니다</strong>. 관측된 데이터를 기반으로 <strong>robust한 standard error</strong>을 계산하는데요, 이를 <strong>sandwich estimator</strong>라고 부릅니다. 이 방식을 사용하게 되면 분포 가정에 있어 비교적 자유롭고, 설령 설정한 상관구조가 조금 잘못되었더라도 비교적 정확한 표준 오차를 제공합니다.</p>
</section><section id="blue-vs-blup" class="level2"><h2 class="anchored" data-anchor-id="blue-vs-blup">BLUE vs BLUP</h2>
<p><strong>BLUE(Best Linear Unbiased Estimator)</strong>: fixed effects에 대한 추정량입니다. 우리가 알고자 하는 모집단 전체의 평균적인 경향성을 나타내는데, 예를 들어 평균적으로 치료 수준을 한 단위 높였을 때 인슐린 농도가 어느 정도 변하는가?에 대한 답입니다. lmer과 GEE 모두 fixed effects가 존재하기에, 두 모델에서 모두 구할 수 있습니다.</p>
<p><strong>BLUP(Best Linear Unbiased Predictor)</strong>: random effects에 대한 <strong>예측치</strong>입니다. BLUE와 다르게 추정이 아니라 예측이라는 점에 주목해야 합니다. BLUE가 모집단 전체의 평균적인 경향성에 주목했다면, BLUP는 모집단 평균에서 벗어난 각 개인의 고유한 편차를 나타냅니다. 예컨대, 3번째 환자는 평균보다 얼마나 더 인슐린 농도가 늘었을까?에 대한 답입니다. random effects에 대한 부분이기에, population-average에 해당하는 GEE에서는 구할 수 없고, lmer에서만 구할 수 있습니다. 한 번 살펴 보겠습니다.</p>
<p>증명의 기초는 <strong>Henderson</strong>이 1959년에 발명한 <strong>Mixed Model Equations(MME)</strong>를 따라갑니다. 증명이 너무 길고 이해가 어려울 수도 있어 복잡한 계산 과정은 과감히 생략하도록 하겠습니다. 행렬형태의 식으로 표현하기 위해, 앞선 노테이션과 조금 다르게, <img src="https://latex.codecogs.com/png.latex?u">의 분산-공분산 행렬을 <img src="https://latex.codecogs.com/png.latex?G">로, <img src="https://latex.codecogs.com/png.latex?%5Cepsilon">의 분산-공분산 행렬은 <img src="https://latex.codecogs.com/png.latex?R">이라고 하겠습니다.</p>
<p>lmer은 likelihood를 기반으로 작동하는 만큼, 이번에는 likelihood로 전개를 해보겠습니다.(사실 LS를 쓰나 Likelihood를 쓰나 결국 똑같은 형태가 나옵니다.) n은 전체 sample size를, r은 random effects의 차원을 의미합니다. 우선 다음과 같이 conditional 형태로 likelihood를 적어봅시다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Af(Y,u)%20=%20f(Y%5Cmid%20u)%20%5Ccdot%20f(u)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cfrac%7B1%7D%7B(%7B2%5Cpi%7D)%5E%5Cfrac%7Bn%7D%7B2%7D%5Cdet%20R%5E%7B1/2%7D%7D%20%5Cexp(-%20%5Cfrac%7B1%7D%7B2%7D(Y-X%5Cbeta-Zu)%5ET%20V%5E%7B-1%7D%5Cfrac%7B1%7D%7B2%7D(Y-X%5Cbeta-Zu))%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctimes%20%5Cfrac%7B1%7D%7B(%7B2%5Cpi%7D)%5E%5Cfrac%7Br%7D%7B2%7D%5Cdet%20G%5E%7B1/2%7D%7D%20%5Cexp(-%20%5Cfrac%7B1%7D%7B2%7Du%5ETGu)%0A"></p>
<p>이제 여기에 로그를 씌우게 되면 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">와 <img src="https://latex.codecogs.com/png.latex?u"> 모두에 대해 2차식으로 볼 수 있습니다. likelihood function의 concave함을 이용해 각각 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">와 <img src="https://latex.codecogs.com/png.latex?u"> 모두로 편미분을 해주고 각각 hat을 씌워주면, 아래와 같이 나타낼 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Balign*%7D%0AX%5ETR%5E%7B-1%7DX%5Chat%7B%5Cbeta%7D%20+%20X%5ETR%5E%7B-1%7DZ%5Chat%7Bu%7D%20&amp;=%20X%5ETR%5E%7B-1%7DY%20%5C%5C%0AZ%5ETR%5E%7B-1%7DX%5Chat%7B%5Cbeta%7D%20+%20(Z%5ETR%5E%7B-1%7DZ+G%5E%7B-1%7D)%5Chat%7Bu%7D%20&amp;=%20Z%5ETR%5E%7B-1%7DY%0A%5Cend%7Balign*%7D%0A"> 이제 이를 행렬 형태로 나타내면, 아래와 같이 쓸 수 있겠습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bbmatrix%7D%0AX%5ET%20R%5E%7B-1%7D%20X%20&amp;%20X%5ET%20R%5E%7B-1%7D%20Z%20%5C%5C%20Z%5ET%20R%5E%7B-1%7D%20X%20&amp;%20Z%5ET%20R%5E%7B-1%7D%20Z%20+%20G%5E%7B-1%7D%20%5Cend%7Bbmatrix%7D%20%5Cbegin%7Bbmatrix%7D%20%5Chat%7B%5Cbeta%7D%20%5C%5C%20%5Chat%7Bu%7D%20%5Cend%7Bbmatrix%7D%20=%20%5Cbegin%7Bbmatrix%7D%20X%5ET%20R%5E%7B-1%7D%20Y%20%5C%5C%20Z%5ET%20R%5E%7B-1%7D%20Y%0A%5Cend%7Bbmatrix%7D%0A"> 위 행렬식이 MME의 기본적인 형태입니다. 이를 정말 열심히 열심히 정리하면 다음과 같은 <strong>BLUE</strong>와 <strong>BLUP</strong>를 얻을 수 있습니다. <img src="https://latex.codecogs.com/png.latex?%0ABLUE%20=%20%5Chat%5Cbeta%20=%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7DY:%20%5Ctext%7B%20GLS%20estimator%7D%0A"> <img src="https://latex.codecogs.com/png.latex?%0ABLUP%20=%20%5Chat%20u%20=GZ%5ETV%5E%7B-1%7D(Y-X%5Chat%5Cbeta)%0A"></p>
<p>여기서 한 가지 의문이 들 수 있는데요, 일반적으로 <strong>Gauss-Markov Theorem</strong>을 만족할 때 OLS를 통해 구한 estimator가 <strong>unbiased</strong>하고 <strong>가장 작은 분산</strong>을 갖기에 <strong>BLUE</strong>라고 부릅니다. 반복측정자료는 같은 subject에서 측정된 데이터 간 상관관계가 존재하기 때문에 <strong>일반적으로 Gauss-Markov Theorem을 만족하지 않습니다</strong>. 그렇다면 과연 lmer을 통해 구한 <img src="https://latex.codecogs.com/png.latex?%5Chat%5Cbeta_%7BGLS%7D">는 BLUE가 맞을까요? <strong>놀랍게도 맞습니다.</strong> 간단하게만 알아보면, Least Square을 일반화해서 GLS를 진행한 것처럼, Gauss-Markov Theorem을 일반화한 <strong>Aitken’s Theorem</strong>이 존재합니다! 이 정리를 통해 GLS estimator가 BLUE임을 보일 수 있습니다. 간단한 증명은 다음과 같습니다.</p>
<p>우선 <strong>unbiasedness</strong>부터 보면, <img src="https://latex.codecogs.com/png.latex?%0AE%5B%5Chat%5Cbeta_%7BGLS%7D%5D%20=%20E%5B(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D(X%5Cbeta+%5Cepsilon%5E*)%5D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20E%5B(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7DX%5Cbeta%5D%20+%20E%5B(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D%5Cepsilon%5E*%5D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20E%5B%5Cbeta%5D%20+%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D%20E%5B%5Cepsilon%5E*%5D%20=%20%5Cbeta%0A"> 이렇게 쉽게 unbiasedness를 보일 수 있습니다. 나아가 분산을 보겠습니다. <img src="https://latex.codecogs.com/png.latex?b">를 또다른 linear unbiased estimator로 이면서 다음과 같은 형태를 만족한다고 해봅시다. <img src="https://latex.codecogs.com/png.latex?%0Ab%20=%20%5B(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D%20+%20A%5DY%0A"> unbiasedness를 만족해야 하기 때문에, 계산을 해보면 <img src="https://latex.codecogs.com/png.latex?AX%20=0">이 되어야 합니다. 그렇다면 분산을 구해보면, <img src="https://latex.codecogs.com/png.latex?%0AVar(b)=%20%5B(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D%20+%20A%5D%20V%20%5B(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETV%5E%7B-1%7D%20+%20A%5D%5E%7B-1%7D%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D%20+%20AVA%5ET%20+(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7DX%5ETA%5ET%20+%20AX(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D%20+%20AVA%5ET%20(%5Cbecause%20%5C%20AX%20=%200)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Cgeq%20(X%5ETV%5E%7B-1%7DX)%5E%7B-1%7D%20(%5Cbecause%20%5C%20V%20%5Ctext%7B%20is%20psd%7D)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20Var(%5Chat%5Cbeta_%7BGLS%7D)%0A"></p>
<p>따라서 <img src="https://latex.codecogs.com/png.latex?%5Chat%5Cbeta_%7BGLS%7D">가 BLUE가 됨을 보일 수 있습니다. BLUP 또한 BLUE와 비슷한 방법으로 또다른 linear unbiased parameter을 들고 와서 증명이 가능합니다. BLUP의 형태를 다시 한 번 보겠습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0ABLUP%20=%20%5Chat%20u%20=GZ%5ETV%5E%7B-1%7D(Y-X%5Chat%5Cbeta)%0A"> 이제 이 식을 우리가 계속 다뤘던 intercept에만 random effects가 존재하는 모델에 맞추어 바꿔보도록 하겠습니다. <img src="https://latex.codecogs.com/png.latex?G"> 또한 원래의 노테이션으로 바꿔서 i번째 환자의 입장에서 정리해보면 다음과 같이 표현할 수 있겠습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%20u_i%20=%20(%5Cfrac%7Bn_i%20%5Csigma_u%20%5E2%7D%7Bn_i%20%5Csigma_u%5E2%20+%20%5Csigma%5E2%7D)%20%5Ctimes%20%5Cfrac%7B1%7D%7Bn_i%7D%20%5Csum_%7Bj=1%7D%5E%7Bn_i%7D(y_%7Bij%7D-x_%7Bij%7D%5ET%5Chat%5Cbeta)%0A"> 이렇게 간단한게 스칼라 형태로 정리 되는데요, 이 식이 가지는 의미는 무엇일까요?</p>
<p>우선 뒷부분부터 보면 우리가 평소에 회귀분석에서 볼 수 있는 <strong>잔차의 평균</strong> 형태입니다. i번째 환자가 평균적으로 예측에서 얼마나 벗어나있는지에 대해 알려주는 정보로 해석할 수 있습니다. 또한 일반적인 선형 회귀 모델과 다르게 우리 모델은 <img src="https://latex.codecogs.com/png.latex?Zu">라는 random effects를 표현하는 term이 존재했으므로, 이 부분을 fixed effects의 효과를 제거하고 random effects와 <img src="https://latex.codecogs.com/png.latex?%5Cepsilon">에 해당하는 error만 남긴 부분으로 볼 수도 있습니다.</p>
<p>첫번째 부분이 BLUP의 핵심이라고 볼 수 있습니다. 이는 가중치의 형태로 <strong>수축(Shrinkage) 효과</strong>를 보여주는데요, 직관적으로 <img src="https://latex.codecogs.com/png.latex?n_i">의 값이 커지거나 <img src="https://latex.codecogs.com/png.latex?%5Csigma_u%5E2"> 커질수록 이 값은 1에 가까워집니다. 각 환자의 데이터가 많거나 환자 내에서 측정한 데이터끼리의 상관관계가 강할수록 가중치가 1에 가까워져 각 환자의 평균 잔차를 더 신뢰하는 형태가 됩니다. 반대로, 0에 가까울수록 각 환자의 데이터보다는 전체 평균에 가까워지도록 예측치를 수축시킵니다.</p>
<p>결국 BLUP는 각 환자의 <strong>반복측정된 데이터의 수</strong>와 그 데이터끼리 <strong>상관된 정도</strong>에 따라 어떻게 예측을 진행할지를 결정해준다고 볼 수 있습니다. 데이터가 부족한 상황에서는 예측이 불확실한 노이즈에 과도하게 영향을 받는 것을 막고, 전체 데이터의 정보에 따라 안정적이고 보수적인 예측을 가능하게 해준다는 점에서 의미가 있습니다.</p>
<p>lmer은 모집단의 전반적인 경향성뿐만 아니라 그 안에서 개개인이 얼마나 다양한 패턴을 보여주는지 또한 제시할 수 있다는 점에서 GEE와는 다른 차별점을 가진다고 할 수 있겠습니다.</p>
</section></section><section id="마치며" class="level1"><h1>마치며</h1>
<p>결국 lmer에서 intercept에만 random effects가 존재할 때 추정치들의 표준 오차가 같아지는 현상은 무언가 잘못된 것이 아니라, 모델 구조상 자연스러운 결과입니다. 반복측정자료 연구에서 주된 관심사가 개체간 변동성을 고려하는 즉, subject(neighbor)-specific하다면, 이는 받아들일 수 있는 결과입니다.</p>
<p>다만, 전체 집단의 평균적인 변화에 관심이 있다면 GEE와 같은 population-average 모델이 더 좋은 방법이 될 수 있습니다. random effects의 분포를 가정하지 않으면서 robust한 추정량을 제공하는 GEE가 대안이 될 수도 있습니다.</p>
<p>결국 가장 중요한 것은 <strong>내가 지금 무엇을 하고 있는지에 대해 정확히 아는 것</strong>입니다. 이 질문에 대해 정확히 대답할 수 있어야 이에 따라 적합한 모델과 그 결과에 대한 올바른 해석을 할 수 있을 것입니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{jee2025,
  author = {Jee, Mingu},
  title = {반복측정자료 {모델링:} Linear Mixed Effects Model},
  date = {2025-08-12},
  url = {https://blog.zarathu.com/posts/2025-08-08-lmer and gee/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-jee2025" class="csl-entry quarto-appendix-citeas">
Jee, Mingu. 2025. <span>“반복측정자료 모델링: Linear Mixed Effects
Model.”</span> August 12, 2025. <a href="https://blog.zarathu.com/posts/2025-08-08-lmer and gee/">https://blog.zarathu.com/posts/2025-08-08-lmer
and gee/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <guid>https://blog.zarathu.com/posts/2025-08-08-lmer and gee/</guid>
  <pubDate>Tue, 12 Aug 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/img/LMER.png" medium="image" type="image/png" height="96" width="144"/>
</item>
<item>
  <title>R 패키지 문서에 서버 없이 Shiny 앱 추가하는 법</title>
  <dc:creator>Gyeom Hwangbo</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/</link>
  <description><![CDATA[ 
<!-- Google tag (gtag.js) -->
<script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script>





<section id="참고-글" class="level1">
<h1>참고 글</h1>
<ul>
<li><a href="https://blog.zarathu.com/posts/2023-03-17-pkgdown/index.html">pkgdown</a></li>
<li><a href="https://blog.zarathu.com/posts/2024-03-04-shinylive/index.html#shinylive">shinylive</a></li>
</ul>
</section>
<section id="getting-started" class="level1">
<h1>Getting started</h1>
<p>pkgdown을 이용한 R 패키지 문서화와 서버 없이 작동하는 Shiny 앱 만들기는 위 글에서 확인할 수 있습니다. 이 글에서는 간단한 R package 생성부터 pkgdown으로 gh-pages에 deploy한 웹페이지에 Shiny 앱을 추가하는 방법을 소개합니다.</p>
<section id="shinylive란" class="level4">
<h4 class="anchored" data-anchor-id="shinylive란">Shinylive란?</h4>
<p>“Shinylive”는 Shiny 애플리케이션을 브라우저 안에서 실행할 수 있게 해주는 도구입니다. 전통적인 Shiny는 R 서버가 필요합니다. 반면에 Shinylive는 R과 Shiny 애플리케이션을 브라우저 자체에 탑재합니다.</p>
<p>장점</p>
<ul>
<li>서버 없이 실행 가능해 호스팅 비용, 유지보수가 필요 없습니다.</li>
<li>정적 파일로 Github Pages 같은 곳에 바로 배포 가능합니다.</li>
</ul>
<p>단점</p>
<ul>
<li>초기 로딩 속도가 느립니다.</li>
<li>사용자 기기에서 모든 연산이 이루어져 복잡한 계산은 느릴 수 있습니다.</li>
<li>코드가 브라우저에 노출됩니다.</li>
</ul>
</section>
</section>
<section id="준비물" class="level1">
<h1>준비물</h1>
<ul>
<li>패키지 설치</li>
</ul>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install.packages</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"shiny"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"shinylive"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pkgdown"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"usethis"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"devtools"</span>))</span></code></pre></div></div>
<ul>
<li>R package</li>
<li>Shiny app</li>
</ul>
</section>
<section id="r-package-생성" class="level1">
<h1>R package 생성</h1>
<ol type="1">
<li>빈 git repository(public)를 생성 후 로컬에 clone 합니다.</li>
<li>clone된 폴더로 R package를 생성 합니다.</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">usethis<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">create_package</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"."</span>) <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># R package 생성</span></span></code></pre></div></div>
<ol start="3" type="1">
<li><p>R 함수 생성</p>
<p>R/ 아래에 함수 hello.R을 생성합니다.</p></li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' # hello.R</span></span>
<span id="cb3-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' Say hello</span></span>
<span id="cb3-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#'</span></span>
<span id="cb3-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @param name A character name</span></span>
<span id="cb3-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @return A greeting</span></span>
<span id="cb3-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#' @export</span></span>
<span id="cb3-7">hello <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(name) {</span>
<span id="cb3-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Hello,"</span>, name)</span>
<span id="cb3-9">}</span></code></pre></div></div>
</section>
<section id="shiny-app-생성" class="level1">
<h1>Shiny app 생성</h1>
<ol type="1">
<li>폴더 생성</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Rstudio Terminal에 입력</span></span>
<span id="cb4-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mkdir</span> <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">-p</span> inst/app/source</span>
<span id="cb4-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cp</span> R/hello.R inst/app/source</span></code></pre></div></div>
<ol start="2" type="1">
<li><p>Shiny app 생성</p>
<p>inst/app/ 아래에 app.R을 생성합니다.</p></li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># app.R</span></span>
<span id="cb5-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(shiny)</span>
<span id="cb5-3"></span>
<span id="cb5-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">source</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"source/hello.R"</span>)</span>
<span id="cb5-5"></span>
<span id="cb5-6">ui <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fluidPage</span>(</span>
<span id="cb5-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">textInput</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"name"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Name:"</span>),</span>
<span id="cb5-8">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">textOutput</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"out"</span>)</span>
<span id="cb5-9">)</span>
<span id="cb5-10"></span>
<span id="cb5-11">server <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(input, output, session) {</span>
<span id="cb5-12">  output<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>out <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">renderText</span>({</span>
<span id="cb5-13">    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hello</span>(input<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>name)</span>
<span id="cb5-14">  })</span>
<span id="cb5-15">}</span>
<span id="cb5-16"></span>
<span id="cb5-17"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">shinyApp</span>(ui, server)</span></code></pre></div></div>
</section>
<section id="shinylive-app-생성" class="level1">
<h1>Shinylive app 생성</h1>
<p>pkgdown은 docs라는 폴더를 생성하고 docs 내부에 파일들을 생성합니다. 이 때, pkgdown은 pkgdown/assets/ 아래 있는 모든 파일들을 docs 폴더로 복사합니다. pkgdown/assets/ 아래 Shinylive app을 export합니다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1">shinylive<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">export</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"inst/app"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"pkgdown/assets/site"</span>)</span></code></pre></div></div>
<p>export 후 pkgdown/assets/site/ 아래에 다음과 같은 파일들이 생성됩니다. <img src="https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/img/liveresult.PNG" class="img-fluid" alt="결과 파일들"></p>
</section>
<section id="pkgdown-문서화" class="level1">
<h1>pkgdown 문서화</h1>
<ol type="1">
<li>문서화</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1">devtools<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">document</span>()</span>
<span id="cb7-2">usethis<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_pkgdown_github_pages</span>()</span>
<span id="cb7-3">usethis<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">use_pkgdown</span>()</span>
<span id="cb7-4">pkgdown<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">build_site</span>()</span></code></pre></div></div>
<p>docs/ 아래에 다음과 같은 파일들이 생성됩니다. <img src="https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/img/pkgresult.PNG" class="img-fluid" alt="결과 파일들"></p>
<ol start="2" type="1">
<li><p>_pkgdown.yml 수정</p>
<p>_pkgdown.yml 파일의 navbar에 Shinylive app 링크를 추가합니다. 기존 pkgdown 문서 아래에 다음 내용을 추가합니다.</p></li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode yml code-with-copy"><code class="sourceCode yaml"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">navbar</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb8-2"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">  </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">structure</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb8-3"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">left</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb8-4"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">-</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> home</span></span>
<span id="cb8-5"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">-</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> reference</span></span>
<span id="cb8-6"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">-</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> hello</span></span>
<span id="cb8-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">  </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">components</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb8-8"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hello</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb8-9"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">      </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">text</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> Hello</span></span>
<span id="cb8-10"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">      </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">href</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> site/index.html</span></span></code></pre></div></div>
<p>이후 다시 build_site()를 해봅니다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1">pkgdown<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">build_site</span>()</span></code></pre></div></div>
<p>여기서 navbar에 Hello 탭이 생기는 것을 확인할 수 있습니다. (여기서 Hello 동작 안하지만 정상적인 현상입니다.) .gitignore 파일에서 docs를 제거 후 git에 commit합니다. github repository에 gh-pages가 생성됩니다. (약간의 시간이 소요됩니다.) 연결된 gh-pages에 들어가면 Hello가 정상적으로 동작 하는 것을 확인할 수 있습니다.</p>
</section>
<section id="정리" class="level1">
<h1>정리</h1>
<p>수정 및 생성한 폴더와 파일은 다음과 같습니다.</p>
<pre><code>myPackage/
├── _pkgdown.yml
├── .gitignore
├── inst/
│   └── app/
│       ├── app.R
│       └── source/
│           └── hello.R
└── R/
    └── hello.R</code></pre>
</section>
<section id="ui-tweak" class="level1">
<h1>UI tweak</h1>
<p>이렇게 추가한 Shinylive app은 UI 구성 요소가 다르다는 문제가 있습니다. UI를 pkgdown 문서와 일치시키는 방법을 소개합니다.</p>
<p>pkgdown으로 생성된 문서의 UI는 docs/index.html과 이에 연결된 .css 및 .js 스크립트들로 이루어져 있습니다. 따라서 index.html을 복사하고, 내용을 지우고, Shinylive app의 내용을 추가하면 같은 UI를 사용할 수 있습니다.</p>
<ol type="1">
<li>docs/index.html을 pkgdown/assets에 복사</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb11-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Rstudio Terminal에 입력</span></span>
<span id="cb11-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cp</span> docs/index.html pkgdown/assets/hello.html</span></code></pre></div></div>
<p>1-1. index.html은 다음과 같은 구조로 되어 있습니다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode html code-with-copy"><code class="sourceCode html"><span id="cb12-1"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;!DOCTYPE</span> html<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-2"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">head</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;!-- .css, .js 스크립트 연결 --&gt;</span></span>
<span id="cb12-4"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">head</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-5"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">body</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-6">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">nav</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-7">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">nav</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-8">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">div</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> class</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"container template-home"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-9">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">div</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> class</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"row"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-10">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">main</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"main"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> class</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"col-md-9"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-11">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">main</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-12">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">aside</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> class</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"col-md-3"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">div</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> class</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"license"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-13">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">aside</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-14">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">div</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-15">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">footer</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-16">      <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">footer</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-17">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">div</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-18"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">body</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb12-19"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">html</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span></code></pre></div></div>
<p>각 <code>&lt;nav&gt;</code>, <code>&lt;main&gt;</code>, <code>&lt;aside&gt;</code>, <code>&lt;footer&gt;</code>는 아래 부분을 정의합니다. <img src="https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/img/area.PNG" class="img-fluid" alt="각 영역 설명"></p>
<ol start="2" type="1">
<li>hello.html에서 사용할 <code>&lt;nav&gt;</code>, <code>&lt;footer&gt;</code>를 남겨두고 <code>&lt;aside&gt;</code>를 삭제합니다. <code>&lt;main&gt;</code> 아래 내용을 아래와 같이 수정합니다.</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="sourceCode html code-with-copy"><code class="sourceCode html"><span id="cb13-1">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">main</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> id</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"main"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> class</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"col-md-12"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> style</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"max-width: 1200px;"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb13-2">    <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">iframe</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> src</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"site/index.html"</span><span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;"> style</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"width:100%; height:calc(100vh - 106px); border:none;"</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">iframe</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb13-3">  <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">main</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span></code></pre></div></div>
<p><code>&lt;main&gt;</code> 영역의 크기를 조절할 때, <code>border:1px solid black;</code>로 테두리 설정하면 조절하기 용이합니다.</p>
<ol start="3" type="1">
<li>_pkgdown.yml의 navbar 부분을 다음과 같이 수정합니다.</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode yml code-with-copy"><code class="sourceCode yaml"><span id="cb14-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">navbar</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb14-2"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">  </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">structure</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb14-3"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">left</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb14-4"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">-</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> home</span></span>
<span id="cb14-5"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">-</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> reference</span></span>
<span id="cb14-6"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">-</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> hello</span></span>
<span id="cb14-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">  </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">components</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb14-8"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">hello</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb14-9"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">      </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">text</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> Hello</span></span>
<span id="cb14-10"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">      </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">href</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> hello.html</span></span></code></pre></div></div>
<ol start="4" type="1">
<li>마지막으로 다시 사이트를 업데이트 하고 git에 commit 합니다.</li>
</ol>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb15-1">pkgdown<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">build_site</span>()</span></code></pre></div></div>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{hwangbo2025,
  author = {Hwangbo, Gyeom},
  title = {R {패키지} {문서에} {서버} {없이} {Shiny} {앱} {추가하는}
    {법}},
  date = {2025-07-29},
  url = {https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-hwangbo2025" class="csl-entry quarto-appendix-citeas">
Hwangbo, Gyeom. 2025. <span>“R 패키지 문서에 서버 없이 Shiny 앱 추가하는
법.”</span> July 29, 2025. <a href="https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/">https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/</a>.
</div></div></section></div> ]]></description>
  <category>shinylive</category>
  <guid>https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/</guid>
  <pubDate>Tue, 29 Jul 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-07-29-pkgdownshiny/img/logo.png" medium="image" type="image/png" height="89" width="144"/>
</item>
<item>
  <title>ggplot으로 만든 그래프에 p-value 추가하는 법</title>
  <dc:creator>Yeji Kang</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-07-28-Pval/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="p-value-시각화" class="level1"><h1>p-value 시각화</h1>
<p>ggplot2로 그룹 간 평균 차이를 시각화할 때, p-value를 함께 표시하면 결과를 보다 직관적으로 이해할 수 있습니다.</p>
<p>이때 흔히 사용하는 함수가 <code>stat_compare_means()</code>인데, 기본적으로 p-value를 자동 계산해 그래프에 추가해주는 유용한 기능을 제공합니다.</p>
<p>다만 이 함수는 <code>p.format</code>이나 <code>p.signif</code> 옵션을 통해 소수점 표기 또는 <code>*</code>, <code>**</code>, <code>***</code> 기호로 유의성을 강조하는 정도만 가능하며, <code>p &lt; 0.05</code>나 <code>p &lt; 0.001</code>처럼 원하는 형식으로 자유롭게 표현하는 데는 한계가 있습니다.</p>
<p>따라서 이번 글에서는 <code>stat_compare_means()</code> 대신 <code>compare_means()</code>로 직접 p-value를 계산한 뒤, 그래프 위에 원하는 형식으로 표시하는 방법을 소개합니다. 특히 <code>p &lt; 0.001</code>과 같은 유의성을 명확하게 전달하고 싶은 경우에 유용한 실전 예제를 중심으로 다뤄보겠습니다.</p>
</section><section id="practice" class="level1"><h1>Practice</h1>
<p>먼저 아래는 필요한 패키지와 함께 예제 데이터를 생성하는 코드입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://r-datatable.com">data.table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://ggplot2.tidyverse.org">ggplot2</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://rpkgs.datanovia.com/ggpubr/">ggpubr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2025</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  ID <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>, times <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Group <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Visit <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Baseline"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Week 2"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Week 4"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, times <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Score <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">70</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">68</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">66</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">70</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">55</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>코드를 통해 생성된 데이터는 다음과 같습니다. 이는 세 시점(Baseline, Week 2, Week 4)에 걸쳐 두 그룹 간의 점수를 비교하기 위한 목적으로 구성된 데이터입니다.</p>
<div class="cell">
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead><tr class="header">
<th style="text-align: right;">ID</th>
<th style="text-align: left;">Group</th>
<th style="text-align: left;">Visit</th>
<th style="text-align: right;">Score</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: right;">1</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">72.48303</td>
</tr>
<tr class="even">
<td style="text-align: right;">2</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">70.14257</td>
</tr>
<tr class="odd">
<td style="text-align: right;">3</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">73.09262</td>
</tr>
<tr class="even">
<td style="text-align: right;">4</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">75.08996</td>
</tr>
<tr class="odd">
<td style="text-align: right;">5</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">71.48390</td>
</tr>
<tr class="even">
<td style="text-align: right;">6</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">69.34858</td>
</tr>
<tr class="odd">
<td style="text-align: right;">7</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">71.58845</td>
</tr>
<tr class="even">
<td style="text-align: right;">8</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">69.68004</td>
</tr>
<tr class="odd">
<td style="text-align: right;">9</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">68.62014</td>
</tr>
<tr class="even">
<td style="text-align: right;">10</td>
<td style="text-align: left;">A</td>
<td style="text-align: left;">Baseline</td>
<td style="text-align: right;">72.80861</td>
</tr>
</tbody>
</table>
</div>
</div>
<p>Visit 변수의 순서를 Baseline → Week 2 → Week 4로 지정해 그래프에서 시간 순서대로 정렬되도록 합니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Visit</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Visit</span>, levels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Baseline"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Week 2"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Week 4"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span></code></pre></div></div>
</div>
<p>다음으로, 각 Visit 시점별로 두 그룹 간 평균 차이에 대한 t-test를 수행합니다. <code>compare_means()</code> 함수를 사용하면, <code>group.by = "Visit"</code> 옵션을 통해 각 시점별로 그룹 간 비교가 이뤄지며, 그 결과가 데이터프레임 형태로 저장됩니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pval_df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">compare_means</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Score</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>, group.by <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Visit"</span>, method <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"t.test"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><code>pval_df</code>에는 각 Visit에 해당하는 시점별로 t-test 결과가 행 단위로 저장되며, 그 중 <code>pval_df$p</code> 열에는 계산된 p-value 값이 포함되어 있습니다.</p>
<p>이제 계산된 p-value를 시각적으로 명확하게 전달될 수 있게 만듭니다. <code><a href="https://rdrr.io/r/base/ifelse.html">ifelse()</a></code> 함수를 사용하여 p-value가 0.001보다 작을 경우 “p &lt; 0.001”로, 그렇지 않으면 소수점 셋째 자리까지 표기하도록 합니다.</p>
<p>이렇게 만든 텍스트 라벨은 pval_df$label이라는 새로운 열로 저장됩니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pval_df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">label</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pval_df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.001</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p &lt; 0.001"</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"p = %.3f"</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pval_df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>이 <code>pval_df$label</code>은 이후 <code>geom_text()</code>에서 <code>label = label</code>로 호출되어 그래프에 p-value 텍스트로 표시됩니다.</p>
<p>이제 <code>stat_summary()</code>로 mean과 standard error를 line graph로 시각화하고, 앞서 계산한 p-value는 <code>geom_text()</code>를 활용해 각 시점 위에 직접 추가합니다. p-value 텍스트가 겹치지 않도록 적절한 y축 위치도 함께 지정해줍니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pval_df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">85</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># adjust label height</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Visit</span>, y <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Score</span>, color <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span>, group <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Group</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stat_summary</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>fun <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mean</span>, geom <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"line"</span>, size <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">stat_summary</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>fun.data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mean_se</span>, geom <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"errorbar"</span>, width <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_text</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pval_df</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Visit</span>, y <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>, label <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, inherit.aes <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>여기서 <code>inherit.aes = FALSE</code>를 지정한 이유는 <code>geom_text()</code>에 별도로 지정한 <code>pval_df</code>의 <code>aesthetics(x, y, label)</code>만 사용하고, 기본 <code>ggplot()</code>에 설정된 <code>aes(x = Visit, y = Score, ...)</code>를 상속받지 않도록 하기 위해서입니다.</p>
<p>이 옵션이 없으면 <code>geom_text()</code>가 Score 값을 y로 사용하려 하면서 오류가 나거나 잘못된 위치에 텍스트가 표시될 수 있습니다.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-07-28-Pval/Rplot.png" class="img-fluid figure-img"></p>
<figcaption>Line Graph</figcaption></figure>
</div>
<p>선 위에 표시된 p-value는 각 시점별로 수행한 t-test 결과이며, 특히 p &lt; 0.001과 같이 유의한 차이는 직관적으로 확인할 수 있도록 강조되었습니다. 이런 방식으로 결과 해석을 한눈에 쉽게 할 수 있도록 만들 수 있습니다.</p>
</section><section id="마치며" class="level1"><h1>마치며</h1>
<p>이렇게 <code>compare_means()</code>로 직접 계산한 p-value를 <code>geom_text()</code>를 활용해 원하는 위치에 표시하는 방식은 boxplot과 barplot 등 다양한 ggplot 그래프 유형에도 동일하게 적용할 수 있습니다.</p>
<p>특히 <code>stat_compare_means()</code>의 기본 기능으로는 원하는 형식의 p-value를 표현하기 어려운 경우, 이와 같이 수동적인 방식이 훨씬 유연한 대안이 될 수 있겠습니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{kang2025,
  author = {Kang, Yeji},
  title = {Ggplot으로 {만든} {그래프에} p-Value {추가하는} {법}},
  date = {2025-07-28},
  url = {https://blog.zarathu.com/posts/2025-07-28-Pval/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-kang2025" class="csl-entry quarto-appendix-citeas">
Kang, Yeji. 2025. <span>“Ggplot으로 만든 그래프에 p-Value 추가하는
법.”</span> July 28, 2025. <a href="https://blog.zarathu.com/posts/2025-07-28-Pval/">https://blog.zarathu.com/posts/2025-07-28-Pval/</a>.
</div></div></section></div> ]]></description>
  <category>ggplot</category>
  <guid>https://blog.zarathu.com/posts/2025-07-28-Pval/</guid>
  <pubDate>Mon, 28 Jul 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-07-28-Pval/Rplot.png" medium="image" type="image/png" height="70" width="144"/>
</item>
<item>
  <title>Breslow 방식을 이용한 Cox 예측 모델 만들기</title>
  <dc:creator>Sungho Choi</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-05-07-Breslow/</link>
  <description><![CDATA[ 
<!-- Google tag (gtag.js) -->
<script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script>





<section id="cox-모델이란" class="level2">
<h2 class="anchored" data-anchor-id="cox-모델이란">Cox 모델이란 ?</h2>
<section id="cox-모델의-등장" class="level3">
<h3 class="anchored" data-anchor-id="cox-모델의-등장">Cox 모델의 등장</h3>
<p>Cox 모델은 위험 함수의 구조를 다음과 같이 정의했습니다. <img src="https://latex.codecogs.com/png.latex?%0Ah(t%E2%88%A3X)=h_0%20(t)%20%E2%8B%85%20%5Cexp%E2%81%A1(%5Cbeta%5ET%20X)%0A"></p>
<p>이 식을 하나하나 풀어보면,</p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?h(t%E2%88%A3X)">: 시간 <img src="https://latex.codecogs.com/png.latex?t">에서 공변량 <img src="https://latex.codecogs.com/png.latex?X">를 가진 사람의 위험도</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?h_0(t)">: baseline hazard function<br>
→ 공변량이 모두 0일 때의 ‘기본 위험’ (시간에 따라 변함)</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cexp(%5Cbeta%5ET%20X)">: 변수 X가 위험도에 미치는 영향 (비례 계수)</p></li>
</ul>
<p>이 구조가 바로 <strong>Cox 비례위험모형</strong>입니다.</p>
</section>
<section id="cox-모델의-핵심-특징" class="level3">
<h3 class="anchored" data-anchor-id="cox-모델의-핵심-특징">Cox 모델의 핵심 특징</h3>
<ol type="1">
<li><p>baseline hazard <img src="https://latex.codecogs.com/png.latex?h_0(t)">는 자유롭게 놔둠 → Cox 모델은 시간에 따른 위험 패턴을 미리 정하지 않습니다.</p></li>
<li><p>회귀계수 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">만 추정 → 변수(예: 나이, 성별, 질병 유무)가 사건 발생 속도에 어떻게 영향을 주는 지를 파악합니다.</p></li>
<li><p>위험비를 바로 해석할 수 있음 <img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7Bh(t%E2%88%A3X_1)%7D%7Bh(t%E2%88%A3X_2)%7D=%20%5Cexp%E2%81%A1(%5Cbeta%5ET(X_1%E2%88%92X_2))%0A"> → 두 집단 간 위험률의 비율이 시간과 관계없이 일정하게 유지됩니다.</p></li>
</ol>
</section>
<section id="cox-모델의-결과-해석" class="level3">
<h3 class="anchored" data-anchor-id="cox-모델의-결과-해석">Cox 모델의 결과 해석</h3>
<p>보통 결과는 아래와 같은 표로 나타납니다.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>변수</th>
<th>계수 β</th>
<th>exp(β) (위험비)</th>
<th>해석</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>나이</td>
<td>0.04</td>
<td>1.04</td>
<td>나이가 1세 많을수록 위험 4% 증가</td>
</tr>
<tr class="even">
<td>성별(남성)</td>
<td>0.65</td>
<td>1.91</td>
<td>남성의 위험이 여성보다 약 2배</td>
</tr>
</tbody>
</table>
<p>해석의 중심은 hazard ratio (위험비)에 있습니다.<br>
즉, “이 요인이 위험을 얼마나 증가시키는가?”를 정량적으로 보여주는 것이 Cox 모델의 핵심 목적입니다.</p>
</section>
<section id="예측이-안-되는-이유" class="level3">
<h3 class="anchored" data-anchor-id="예측이-안-되는-이유">예측이 안 되는 이유</h3>
<p>Cox 모델은 설계 자체가 <strong>비교</strong>에 초점이 맞춰져 있습니다. 회귀계수 <img src="https://latex.codecogs.com/png.latex?%5Cbeta"> 추정에는 baseline hazard <img src="https://latex.codecogs.com/png.latex?h_0%20(t)">가 필요하지 않기 때문입니다. 하지만 개별 생존 확률 <img src="https://latex.codecogs.com/png.latex?S(t%7CX)">를 계산하려면 <img src="https://latex.codecogs.com/png.latex?%0AS(t%7CX)%20=%20%5Cexp%20(-H_0%20(t)%20%E2%8B%85%20%5Cexp(%5Cbeta%20%5ET%20X))%0A"> 와 같이 계산해야 하는데, 이 수식에는 baseline hazard가 필수적입니다. 따라서, cox 모델은 baseline hazard를 직접 추정하지 않기 때문에 예측이 불가능했습니다.</p>
</section>
</section>
<section id="baseline-hazard는-어떻게-추정할까" class="level2">
<h2 class="anchored" data-anchor-id="baseline-hazard는-어떻게-추정할까">baseline hazard는 어떻게 추정할까?</h2>
<p>앞서 본 것처럼, Cox 모델은 회귀계수 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">만 추정하고, <strong>baseline hazard</strong> <img src="https://latex.codecogs.com/png.latex?h_0(t)">는 남겨둡니다. 이것은 Cox 모델의 유연성과 강점이기도 하지만, 반대로 예측에는 반드시 <img src="https://latex.codecogs.com/png.latex?H_0(t)"> (누적 baseline hazard)가 필요합니다.</p>
<section id="breslow-방식-가장-널리-쓰이는-추정-방법" class="level3">
<h3 class="anchored" data-anchor-id="breslow-방식-가장-널리-쓰이는-추정-방법">Breslow 방식: 가장 널리 쓰이는 추정 방법</h3>
<p><strong>Norman Breslow</strong>는 1972년 Cox 논문에 대한 토론에서 baseline hazard를 추정할 수 있는 간단한 수식을 제안했습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7BH%7D_0(t)%20=%20%5Csum_%7Bi:%20t_i%20%5Cleq%20t%7D%20%5Cfrac%7Bd_i%7D%7B%5Csum_%7Bj%20%5Cin%20R(t_i)%7D%20%5Cexp%5Cleft(%20%5Chat%7B%5Cbeta%7D%5E%5Ctop%20X_j%20%5Cright)%7D%0A"></p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>기호</th>
<th>의미</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><img src="https://latex.codecogs.com/png.latex?t_i"></td>
<td>사건이 발생한 시점</td>
</tr>
<tr class="even">
<td><img src="https://latex.codecogs.com/png.latex?d_i"></td>
<td>그 시점에 발생한 사건 수</td>
</tr>
<tr class="odd">
<td><img src="https://latex.codecogs.com/png.latex?R(t_i)"></td>
<td>해당 시점에 생존해 있는 사람들 (위험집합)</td>
</tr>
<tr class="even">
<td><img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D"></td>
<td>Cox 모델로 추정한 회귀계수</td>
</tr>
<tr class="odd">
<td><img src="https://latex.codecogs.com/png.latex?X_j"></td>
<td>위험집합 내 개별 환자의 공변량 값</td>
</tr>
</tbody>
</table>
</section>
<section id="breslow-방식의-핵심-아이디어" class="level3">
<h3 class="anchored" data-anchor-id="breslow-방식의-핵심-아이디어">Breslow 방식의 핵심 아이디어</h3>
<blockquote class="blockquote">
<p>사건이 많이 발생한 시간은 위험도가 높고, 위험도가 높은 사람은 더 많이 위험에 기여한다</p>
</blockquote>
<p>그래서 Breslow 방식은 사건 수를 <strong>위험도 총합</strong>에 따라 분배합니다.</p>
<ul>
<li>위 식의 분자는 사건 수</li>
<li>분모는 살아있는 사람들의 위험도 총합</li>
</ul>
<p>→ 이걸 시간 순으로 누적하면 <img src="https://latex.codecogs.com/png.latex?H_0(t)">가 됩니다.</p>
</section>
<section id="직관적으로-정리해-보면" class="level3">
<h3 class="anchored" data-anchor-id="직관적으로-정리해-보면">직관적으로 정리해 보면</h3>
<ul>
<li><p>어떤 시간에 1명이 사망했다면, 그건 그 시점의 <strong>전체 위험도</strong> 중 한 부분이 사라진 것입니다.</p></li>
<li><p>위험도가 높았던 사람일수록 사건 발생이 그 사람 때문일 가능성이 크다고 봅니다.</p></li>
<li><p>그래서 hazard는</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Ctext%7B%EC%82%AC%EA%B1%B4%20%EC%88%98%7D%20%5Cdiv%20%5Ctext%7B%EA%B7%B8%20%EC%8B%9C%EC%A0%90%20%EC%9C%84%ED%97%98%EB%8F%84%20%EC%B4%9D%ED%95%A9%7D"></p>
<p>으로 계산합니다.</p></li>
</ul>
</section>
<section id="손으로-계산해보는-breslow-방식" class="level3">
<h3 class="anchored" data-anchor-id="손으로-계산해보는-breslow-방식">손으로 계산해보는 Breslow 방식</h3>
<section id="예제-데이터" class="level4">
<h4 class="anchored" data-anchor-id="예제-데이터">예제 데이터</h4>
<table class="caption-top table">
<thead>
<tr class="header">
<th>ID</th>
<th>나이(X)</th>
<th>생존 시간(time)</th>
<th>사건 여부(status)</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>A</td>
<td>60</td>
<td>5</td>
<td>1 (사망)</td>
</tr>
<tr class="even">
<td>B</td>
<td>65</td>
<td>7</td>
<td>1 (사망)</td>
</tr>
<tr class="odd">
<td>C</td>
<td>70</td>
<td>7</td>
<td>1 (사망)</td>
</tr>
<tr class="even">
<td>D</td>
<td>55</td>
<td>9</td>
<td>0 (검열)</td>
</tr>
<tr class="odd">
<td>E</td>
<td>62</td>
<td>9</td>
<td>1 (사망)</td>
</tr>
</tbody>
</table>
<ul>
<li><p>공변량은 <strong>나이(X)</strong> 하나만 사용합니다.</p></li>
<li><p>Cox 회귀 계수 <img src="https://latex.codecogs.com/png.latex?%5Chat%20%5Cbeta%20=%200.05"> 라고 가정합니다.</p></li>
</ul>
</section>
<section id="단계-각-환자의-위험도-계산" class="level4">
<h4 class="anchored" data-anchor-id="단계-각-환자의-위험도-계산">1단계: 각 환자의 위험도 계산</h4>
<table class="caption-top table">
<thead>
<tr class="header">
<th>ID</th>
<th>나이</th>
<th>위험도 <img src="https://latex.codecogs.com/png.latex?%5Cexp(0.05%20%5Ccdot%20X)"></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>A</td>
<td>60</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cexp(3.0)%20%5Capprox%2020.09"></td>
</tr>
<tr class="even">
<td>B</td>
<td>65</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cexp(3.25)%20%5Capprox%2025.79"></td>
</tr>
<tr class="odd">
<td>C</td>
<td>70</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cexp(3.5)%20%5Capprox%2033.12"></td>
</tr>
<tr class="even">
<td>D</td>
<td>55</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cexp(2.75)%20%5Capprox%2015.64"></td>
</tr>
<tr class="odd">
<td>E</td>
<td>62</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cexp(3.1)%20%5Capprox%2022.20"></td>
</tr>
</tbody>
</table>
</section>
<section id="단계-사건-발생-시점-정리" class="level4">
<h4 class="anchored" data-anchor-id="단계-사건-발생-시점-정리">2단계: 사건 발생 시점 정리</h4>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?t=5"> : A 사망 (1건)</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?t=7"> : B, C 사망 (2건)</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?t=9"> : E 사망 (1건)</p></li>
</ul>
</section>
<section id="단계-각-시점에서의-hazard-기여-계산" class="level4">
<h4 class="anchored" data-anchor-id="단계-각-시점에서의-hazard-기여-계산">3단계: 각 시점에서의 hazard 기여 계산</h4>
<section id="시간-5" class="level5">
<h5 class="anchored" data-anchor-id="시간-5"><strong>시간 = 5</strong></h5>
<ul>
<li><p>위험집합 <img src="https://latex.codecogs.com/png.latex?R(5)"> : A, B, C, D, E (모두 생존 중)</p></li>
<li><p>총 위험도 합: <img src="https://latex.codecogs.com/png.latex?20.09+25.79+33.12+15.64+22.20=116.84"></p></li>
<li><p>사건 수: 1</p></li>
<li><p>해당 시점의 기여:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5CDelta%20H_0(5)%20=%20%5Cfrac%7B1%7D%7B116.84%7D%20%5Capprox%200.00856"></p></li>
</ul>
</section>
<section id="시간-7" class="level5">
<h5 class="anchored" data-anchor-id="시간-7"><strong>시간 = 7</strong></h5>
<ul>
<li><p>위험집합 <img src="https://latex.codecogs.com/png.latex?R(7)"> : B, C, D, E (A는 사망)</p></li>
<li><p>총 위험도 합: <img src="https://latex.codecogs.com/png.latex?25.79+33.12+15.64+22.20=96.75"></p></li>
<li><p>사건 수: 2</p></li>
<li><p>해당 시점의 기여:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5CDelta%20H_0(7)%20=%20%5Cfrac%7B2%7D%7B96.75%7D%20%5Capprox%200.02068"></p></li>
</ul>
</section>
<section id="시간-9" class="level5">
<h5 class="anchored" data-anchor-id="시간-9"><strong>시간 = 9</strong></h5>
<ul>
<li><p>위험집합 <img src="https://latex.codecogs.com/png.latex?R(9)"> : D, E (B, C는 사망)</p></li>
<li><p>총 위험도 합: <img src="https://latex.codecogs.com/png.latex?15.64+22.20=37.84"></p></li>
<li><p>사건 수: 1</p></li>
<li><p>해당 시점의 기여:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5CDelta%20H_0(9)%20=%20%5Cfrac%7B1%7D%7B37.84%7D%20%5Capprox%200.02643"></p></li>
</ul>
</section>
</section>
<section id="단계-누적-hazard-계산" class="level4">
<h4 class="anchored" data-anchor-id="단계-누적-hazard-계산">4단계: 누적 hazard 계산</h4>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%20H_0%20(5)%20=%200.00856%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%20H_0%20(7)%20=%200.00856%20+%200.02068%20=%200.02924%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%20H_0%20(9)%20=%200.02924%20+%200.02643%20=%200.5567%0A"></p>
</section>
<section id="단계-생존확률-계산-예시-예-d" class="level4">
<h4 class="anchored" data-anchor-id="단계-생존확률-계산-예시-예-d">5단계: 생존확률 계산 예시 (예: D)</h4>
<ul>
<li><p>D의 위험도: <img src="https://latex.codecogs.com/png.latex?%5Cexp(0.05%20%5Ccdot%2055)%20=%20%5Cexp(2.75)%20%5Capprox%2015.64"></p></li>
<li><p>누적 hazard: <img src="https://latex.codecogs.com/png.latex?%5Chat%7BH%7D_0(9)%20=%200.05567"></p></li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?%0AS(9%7CD)%20=%20%5Cexp(-0.05567%20%5Ccdot%2015.64)%20=%20%5Cexp(-0.871)%20%5Capprox%200.418%0A"></p>
<p>→ D가 9일까지 생존할 확률은 약 <strong>41.8%</strong></p>
</section>
</section>
</section>
<section id="실전-r-코드-cox-모델로-생존-예측-모델-만들기" class="level2">
<h2 class="anchored" data-anchor-id="실전-r-코드-cox-모델로-생존-예측-모델-만들기">[실전 R 코드] Cox 모델로 생존 예측 모델 만들기</h2>
<p>앞서 배운 Breslow 방식으로 Baseline Hazard를 구해, 특정 환자 그룹의 생존 확률을 예상해 볼 수 있는 모델을 만들 수 있습니다. 아래는 그 예시입니다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(survival)</span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data</span>(lung)</span>
<span id="cb1-3"></span>
<span id="cb1-4">fit <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coxph</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Surv</span>(time, status) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> age <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> sex, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> lung)</span>
<span id="cb1-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(fit)</span>
<span id="cb1-6"></span>
<span id="cb1-7">basehaz_df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">basehaz</span>(fit, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">centered =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span>
<span id="cb1-8"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">head</span>(basehaz_df)</span>
<span id="cb1-9"></span>
<span id="cb1-10">new_patient <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">age =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">70</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sex =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>)</span>
<span id="cb1-11">lp <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">predict</span>(fit, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">newdata =</span> new_patient, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"lp"</span>) </span>
<span id="cb1-12"></span>
<span id="cb1-13">basehaz_df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>surv <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>basehaz_df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>hazard <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(lp))</span>
<span id="cb1-14"></span>
<span id="cb1-15"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">plot</span>(basehaz_df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>time, basehaz_df<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>surv, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"l"</span>,</span>
<span id="cb1-16">     <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">xlab =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Time"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ylab =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Survival Probability"</span>,</span>
<span id="cb1-17">     <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">main =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Predicted Survival Curve for New Patient"</span>)</span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.zarathu.com/posts/2025-05-07-Breslow/img/new curve.png" class="img-fluid figure-img"></p>
<figcaption>생존 예측 모델</figcaption>
</figure>
</div>
</section>
<section id="더-알아보기-baseline-hazard를-추정하는-다양한-방식" class="level2">
<h2 class="anchored" data-anchor-id="더-알아보기-baseline-hazard를-추정하는-다양한-방식">[더 알아보기] baseline hazard를 추정하는 다양한 방식</h2>
<p>앞서 우리는 <strong>Breslow 방식</strong>으로 baseline hazard <img src="https://latex.codecogs.com/png.latex?H_0(t)">를 추정했습니다. 가장 널리 쓰이는 방법이기도 합니다. 하지만 실제 데이터는 그렇게 단순하지 않습니다.</p>
<p>예를 들어, <strong>동시에 여러 사건이 발생하는 경우</strong>에는 Breslow 방식이 정확하지 않을 수 있습니다.</p>
<p>그래서 여러 가지 방식들이 제안되었고, R에서는 <code>coxph()</code> 함수의 <code>ties</code> 인자를 통해 이를 선택할 수 있습니다.</p>
<section id="왜-다른-방식이-필요한가" class="level3">
<h3 class="anchored" data-anchor-id="왜-다른-방식이-필요한가">왜 다른 방식이 필요한가?</h3>
<blockquote class="blockquote">
<p>현실 데이터에서는 여러 명이 동일한 시간에 이탈하는 일이 자주 발생합니다. 이런 경우, 누가 먼저였는지 모르기 때문에 동시 사건 처리 방식이 중요합니다.</p>
</blockquote>
</section>
<section id="주요-방식-비교" class="level3">
<h3 class="anchored" data-anchor-id="주요-방식-비교">주요 방식 비교</h3>
<table class="caption-top table">
<colgroup>
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
</colgroup>
<thead>
<tr class="header">
<th>방식</th>
<th>특징</th>
<th>동시사건 처리</th>
<th><code>coxph()</code> 옵션</th>
<th>사용 추천 상황</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td><strong>Breslow</strong></td>
<td>기본값, 계산 간단</td>
<td>사건 수를 평균 분배</td>
<td><code>"breslow"</code> (기본값)</td>
<td>동시사건이 적을 때</td>
</tr>
<tr class="even">
<td><strong>Efron</strong></td>
<td>보정된 평균 방식</td>
<td>순차적으로 분배</td>
<td><code>"efron"</code></td>
<td>동시사건이 많을 때</td>
</tr>
<tr class="odd">
<td><strong>Exact</strong></td>
<td>순열 기반 정밀 추정</td>
<td>완전 계산</td>
<td><code>"exact"</code></td>
<td>표본이 작고 정확성이 중요한 경우</td>
</tr>
</tbody>
</table>
</section>
<section id="breslow-vs-efron의-핵심-차이" class="level3">
<h3 class="anchored" data-anchor-id="breslow-vs-efron의-핵심-차이">Breslow vs Efron의 핵심 차이</h3>
<section id="breslow-방식" class="level4">
<h4 class="anchored" data-anchor-id="breslow-방식">Breslow 방식</h4>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?d">건의 사건이 같은 시간에 발생하면, 그냥 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Bd%7D%7B%5Csum%20%5Cexp(%5Cbeta%5ET%20X)%7D">과 같이 평균 분배</p></li>
<li><p>계산은 빠르지만, bias가 생길 수 있음</p></li>
</ul>
</section>
<section id="efron-방식" class="level4">
<h4 class="anchored" data-anchor-id="efron-방식">Efron 방식</h4>
<ul>
<li><p>사건을 <strong>하나씩 순차적으로 발생한 것처럼 나눠서</strong> 분자와 분모를 보정</p></li>
<li><p>보다 현실적인 근사 방식</p></li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CDelta%20H_0(t)%20=%20%5Csum_%7Bk=1%7D%5E%7Bd%7D%20%5Cfrac%7B1%7D%7B%5Csum_%7Bj%20%5Cin%20R(t)%7D%20%5Cexp(%5Cbeta%5ET%20X_j)%20-%20%5Cfrac%7Bk%20-%201%7D%7Bd%7D%20%5Csum_%7Bj%20%5Cin%20D(t)%7D%20%5Cexp(%5Cbeta%5ET%20X_j)%7D%0A"></p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?D(t)"> : 사건이 발생한 사람들</p></li>
<li><p>순차적으로 사건을 일으키면서 분모를 조정함</p></li>
</ul>
</section>
</section>
<section id="r-코드-예시-efron-방식-비교" class="level3">
<h3 class="anchored" data-anchor-id="r-코드-예시-efron-방식-비교">R 코드 예시 : Efron 방식 비교</h3>
<section id="코드-예시" class="level4">
<h4 class="anchored" data-anchor-id="코드-예시">코드 예시</h4>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(survival)</span>
<span id="cb2-2"></span>
<span id="cb2-3">fit_breslow <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coxph</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Surv</span>(time, status) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> age <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> sex, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> lung, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ties =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"breslow"</span>)</span>
<span id="cb2-4">fit_efron   <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coxph</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Surv</span>(time, status) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> age <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> sex, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> lung, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ties =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"efron"</span>)</span>
<span id="cb2-5"></span>
<span id="cb2-6"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 두 모델의 회귀계수 비교</span></span>
<span id="cb2-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(fit_breslow)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>coefficients</span>
<span id="cb2-8"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(fit_efron)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>coefficients</span></code></pre></div></div>
</section>
<section id="코드-결과" class="level4">
<h4 class="anchored" data-anchor-id="코드-결과">코드 결과</h4>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(fit_breslow)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>coefficients</span>
<span id="cb3-2">           coef <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(coef)    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">se</span>(coef)         z    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Pr</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span><span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">|</span>z<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span>)</span>
<span id="cb3-3">age  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.01701289</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.0171584</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.009221954</span>  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.844825</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.065063023</span></span>
<span id="cb3-4">sex <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.51256479</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5989574</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.167462063</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3.060782</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.002207601</span></span>
<span id="cb3-5"><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summary</span>(fit_efron)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>coefficients</span>
<span id="cb3-6">           coef <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(coef)    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">se</span>(coef)         z    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Pr</span>(<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span><span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">|</span>z<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span>)</span>
<span id="cb3-7">age  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.01704533</span>  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.017191</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.009223273</span>  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.848078</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.064591012</span></span>
<span id="cb3-8">sex <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.51321852</span>  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.598566</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.167457962</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3.064760</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.002178445</span></span></code></pre></div></div>
<blockquote class="blockquote">
<p>대부분의 경우 큰 차이는 없지만, 동시 사건이 많거나 위험도가 비슷한 경우에는 의미 있는 차이가 생깁니다.</p>
</blockquote>
</section>
</section>
<section id="기타-방식-exact-kalbfleisch-prentice" class="level3">
<h3 class="anchored" data-anchor-id="기타-방식-exact-kalbfleisch-prentice">기타 방식: Exact, Kalbfleisch-Prentice</h3>
<ul>
<li><p><strong>Exact</strong> 방식은 모든 가능한 사건 순서를 계산하여 우도를 정확히 계산<br>
→ 계산 비용이 높음, 작은 데이터셋에만 적합</p></li>
<li><p><strong>Kalbfleisch-Prentice</strong> 방식은 discrete-time hazard 추정용<br>
→ R 기본에서는 잘 사용되지 않음</p></li>
</ul>


</section>
</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{choi2025,
  author = {Choi, Sungho},
  title = {Breslow {방식을} {이용한} {Cox} {예측} {모델} {만들기}},
  date = {2025-05-07},
  url = {https://blog.zarathu.com/posts/2025-05-07-Breslow/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-choi2025" class="csl-entry quarto-appendix-citeas">
Choi, Sungho. 2025. <span>“Breslow 방식을 이용한 Cox 예측 모델
만들기.”</span> May 7, 2025. <a href="https://blog.zarathu.com/posts/2025-05-07-Breslow/">https://blog.zarathu.com/posts/2025-05-07-Breslow/</a>.
</div></div></section></div> ]]></description>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-05-07-Breslow/</guid>
  <pubDate>Wed, 07 May 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-05-07-Breslow/img/new curve.png" medium="image" type="image/png" height="65" width="144"/>
</item>
<item>
  <title>FDA의 RCT Guidance for industry</title>
  <dc:creator>Hyungwoo Jo</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="시작하기-전에" class="level1"><h1>시작하기 전에</h1>
<p>이 문서의 경우 <a href="https://www.fda.gov/regulatory-information/search-fda-guidance-documents/adjusting-covariates-randomized-clinical-trials-drugs-and-biological-products"><code>Adjusting for Covariates in Randomized Clinical Trials for Drugs and Biological Products Guidance for Industry</code></a>를 기반으로 작성되었습니다.</p>
<section id="intro-is-rct-always-perfect" class="level2"><h2 class="anchored" data-anchor-id="intro-is-rct-always-perfect">Intro: Is RCT always perfect?</h2>
<p>RCT의 경우 대부분 Internal validity의 측면에선 완전하기 때문에, 별다른 공변량의 보정 없이 보고 싶은 변수(ex. 치료 여부 와 결과 여부)만 분석 모형에 포함한 이후에 분석을 하게 됩니다. 다만, 이런 RCT의 경우에도 통계적으로 완전하지 않을 수 있습니다.</p>
<section id="random-imbalance" class="level3"><h3 class="anchored" data-anchor-id="random-imbalance">Random imbalance</h3>
<p>RCT에 참여하게 된 환자를 완전하게 랜덤으로 나누게 되더라도, 통계적으로 두 집단의 특성이 완전히 일치하지 않을 수 있습니다(ex. 한 그룹의 나이가 다른 그룹보다 많을 수 있습니다). 이를 Random imbalance라고 부르는데, 집단의 특성이 여러개인 경우(ex. 나이 성별 당뇨병 유무 등등) 그 중 하나의 특성이라도 차이날 확률을 아래와 같이 계산해볼 수 있습니다 .</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://ggplot2.tidyverse.org">ggplot2</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4000</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">500</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cov_counts</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">probs</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">sapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cov_counts</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">replicate</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covs</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/matrix.html">matrix</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, ncol <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/any.html">any</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">sapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/t.test.html">t.test</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p.value</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>covariates <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cov_counts</span>, prob <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">probs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">covariates</span>, y <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">prob</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_smooth</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>method <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"loess"</span>, span <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.0</span>, se <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_x_continuous</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>breaks <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cov_counts</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Number of Covariates"</span>,</span>
<span>    y <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Probability of ≥1 p-value &lt; 0.05"</span>,</span>
<span>    title <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Chance Imbalance vs Number of Covariates (n = 4000)"</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_y_continuous</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>limits <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/index_files/figure-html/unnamed-chunk-1-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>결과에서 알 수 있듯이, 사람의 수가 4000명이라고 가정하였을 때 보고 싶은 공변량의 갯수가 20개 이상만 되어도 하나 이상의 특성이 두 집단에서 차이가 날 확률이 50프로가 넘게 됩니다. 따라서 RCT에서도 공변량이 많은 경우 두 그룹의 모든 특성이 항상 완전히 일치한다고 말하기는 힘듭니다.</p>
</section><section id="잔차분산" class="level3"><h3 class="anchored" data-anchor-id="잔차분산">잔차분산</h3>
<p>잔차분산(residual variance)은 회귀모델이 설명하지 못한 <strong>남은 변동량</strong>을 나타냅니다.잔차는<img src="https://latex.codecogs.com/png.latex?%5Chat%5Cvarepsilon%5C_i%20=%20y_i%20-%20%5Chat%20y_i"> (관측값 (y_i)에서 모델 예측값 (<img src="https://latex.codecogs.com/png.latex?%5Chat%20y_i">)를 뺀 값)으로 정의되고, 잔차분산 추정치는아래와 같이 계산됩니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%5Csigma%5E2_%7B%5Crm%20res%7D%0A=%20%5Cfrac%7B1%7D%7Bn%20-%20k%7D%20%5Csum_%7Bi=1%7D%5En%20%5Chat%5Cvarepsilon_i%5E2,%0A"></p>
<ul>
<li>(n): 관측치 수<br>
</li>
<li>(k): 회귀계수의 개수(절편 포함)</li>
</ul>
<section id="unadjusted-vs-adjusted-잔차분산-차이" class="level4"><h4 class="anchored" data-anchor-id="unadjusted-vs-adjusted-잔차분산-차이">Unadjusted vs Adjusted 잔차분산 차이</h4>
<p>“치료효과만” 조정한 모형(unadjusted)과 “치료효과 + baseline”을 함께 조정한 모형(adjusted)이 잔차분산에서 어떻게 달라지는지 살펴봅니다.</p>
<p><strong>모델 가정</strong><br><img src="https://latex.codecogs.com/png.latex?%0Ay_i%20=%20%5Calpha%20+%20%5Ctau,T_i%20+%20%5Cbeta,X_i%20+%0A%5Cvarepsilon%5C_i,%5Cquad%20%5Cvarepsilon%5C_i%5Csim%20N(0,%5Csigma%5E2)%0A"> <img src="https://latex.codecogs.com/png.latex?%0AT_i:%20treatment%20%EC%A7%80%EC%8B%9C%EC%9E%90,%20X_i:%20baseline%20%EA%B3%B5%EB%B3%80%EB%9F%89%20(%EC%98%88:%20%EA%B8%B0%EC%A0%80%ED%98%88%EC%95%95),%20%5Cvarepsilon%5C_i:%20%EC%88%9C%EC%88%98%20%EC%98%A4%EC%B0%A8%ED%95%AD%0A"> 이 실제 모델이라고 가정하겠습니다. 즉, baseline에 있는 공변량이 치료 결과에 영향을 미친다고 가정을 하겠습니다(실제 임상에서도 age, DM 유무와 같은 baseline covariate의 경우 치료 결과에 영향을 미치기 때문에 보다 현실적인 가정이라 할 수 있습니다.)</p>
<p>Unadjusted 모형의 경우 아래와 같이 설정할 수 있습니다. <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20%20y_i%20=%20%5Calpha'%20+%20%5Ctau',T_i%20+%20e_i,%20%5Cquad%20e_i%20=%20%5Cbeta,X_i%20+%0A%20%20%20%20%5Cvarepsilon%5C_unadjusted%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%20%20%EC%9E%94%EC%B0%A8%EB%B6%84%EC%82%B0:%20%20%5Cmathrm%7BVar%7D(e_i)%20=%0A%20%20%20%20%7B%5Ctext%7Bbaseline%EC%9D%B4%20%EC%84%A4%EB%AA%85%ED%95%98%EB%8A%94%20%EB%B6%84%EC%82%B0%7D%7D%20+%0A%20%20%20%20%7B%5Ctext%7B%EC%8B%A4%EC%A0%9C%20%EB%B6%84%EC%82%B0%7D%7D.%0A"> 으로 계산이 됨을 알 수 있습니다. Adjusted의 경우 모델이 아래와 같습니다. <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20y_i%20=%20%5Calpha%20+%20%5Ctau,T_i%20+%20%5Cbeta,X_i%20+%20%5Cvarepsilon%5C_adjusted%0A"> 따라서 baseline을 포함하는 경우 실제 잔차에 더 가까운 추정을 할 수 있으며, 이는 잔차분산이 줄어들어 잔차의 <strong>밀도곡선</strong>이 더 좁아지고,결과적으로 treatment 효과의 <strong>표준오차(SE)</strong>가 감소하여 추론이 더 정밀해진다는 것을 알 수 있습니다다. 아래의 R code로 한번 테스트를 해보겠습니다</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://ggplot2.tidyverse.org">ggplot2</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/hadley/reshape">reshape2</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, mean <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">150</span>, sd <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, sd <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>       <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>, treat <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Unadjusted 모형</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span>        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res0</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/residuals.html">resid</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Adjusted 모형(baseline을 포함)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span>        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res1</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/residuals.html">resid</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df_long</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">melt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>,</span>
<span>                   measure.vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"res0"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"res1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>                   variable.name <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"model"</span>,</span>
<span>                   value.name    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"residual"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggplot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df_long</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">aes</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">residual</span>, fill <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">geom_density</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>alpha <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coord_cartesian</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>ylim <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">labs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    title <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unadjusted vs Adjusted 모형에서 잔차분산 비교"</span>,</span>
<span>    x     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"잔차(residual)"</span>,</span>
<span>    y     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"밀도"</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">scale_fill_manual</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    name   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"모형"</span>,</span>
<span>    values <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"res0"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"salmon"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"res1"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"skyblue"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    labels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unadjusted"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Adjusted"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_minimal</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/index_files/figure-html/unnamed-chunk-2-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>결과에서 살펴보실 수 있듯이, 치료군을 랜덤으로 배정한 경우에도 결과에 영향을 미치는 covariate 변수를 보정하였을 때 표준오차가 감소하여 더 정밀한 추론이 가능해집니다.</p>
<p>이제 문서의 내용을 본격적으로 살펴보도록 하겠습니다.</p>
</section></section></section></section><section id="general-considerations" class="level1"><h1>General Considerations</h1>
<p>기본적으로 Unadjusted 분석도 허용을 한다고 되어 있습니다. 하지만 효율성을 위해 예측력이 높은 Covariate들을 사전 지정해 분석 모형에 포함하는 것을 권고한다는 것이 앞으로의 내용이 될 예정입니다. 또한 어떤 방식을 1차로 할지(adjust vs unadjust)는 사전분석계획에 반드시 명시해야한다고 되어있습니다.</p>
<section id="조정할-공변량covariate의-선택" class="level3"><h3 class="anchored" data-anchor-id="조정할-공변량covariate의-선택">조정할 공변량(Covariate)의 선택</h3>
<p>Covariate를 조정하게 되면 앞서 살펴본 바와 같이 variability가 감소되어 신뢰구간이 좁아지고 가설 검정력이 올라가게 됩니다. 다만, Covariate가 결과변수에 영향을 미치는 변수여야efficiency gain이 있습니다. 따라서, 예측력 낮은 Covariate 조정은 도움이 없거나 해가 될 수 있기 때문에 의미 있는 변수들로만 잘 선택하는 것이 중요합니다. 상관관계가 높은 Covariate끼리 조정해도 괜찮지만 (ex. BMI와 weight) 낮은 Covariate들을 선택했을 때 정밀도가 더 올라가게 됩니다.</p>
</section><section id="층화에-사용한-변수도-covariate에-포함하는-것이-좋습니다." class="level3"><h3 class="anchored" data-anchor-id="층화에-사용한-변수도-covariate에-포함하는-것이-좋습니다.">층화에 사용한 변수도 Covariate에 포함하는 것이 좋습니다.</h3>
<p>예를 들어, 당뇨유무로 환자군을 층화하여 랜덤으로 치료군을 배정하였다고 가정해봅시다. 이런 경우에도 최종 모델에는 당뇨변수를 포함하여야 결과가 보다 정확할 수 있습니다. 아래와 같은 예시를 살펴 보겠습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_stratum</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">800</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#dm 유무를 기준으로 층화한 이후에 치료군 배정을 랜덤화</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_stratum</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_stratum</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, times <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_stratum</span>, sd <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>, treatment <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, dm <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#dm 유무를 반영하지 않은 모델</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#dm 유무를 반영한 모델</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">results</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  Model    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unadjusted"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Adjusted"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Estimate <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,</span>
<span>               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  StdError <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,</span>
<span>               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/print.html">print</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">results</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>       Model Estimate  StdError
1 Unadjusted 1.941216 0.1945923
2   Adjusted 1.941216 0.1478912</code></pre>
</div>
</div>
<p>예시의 결과를 살펴보시면, 당뇨의 유무가 최종 치료결과에 영향을 미치게 되기 때문에 이를 반영하였을 떄 표준오차가 줄어드는 것을 알 수 있습니다. (치료군을 랜덤 배정했기 때문에 두 모델 모두 estimate는 실제 값인 2에 가깝게 추정을 하고 있다는 것을 알 수 있습니다) 이와 같이 공변량에 따라 표준오차가 달라질 수 있기때문에, 사용할 공변량을 미리 공시하였을 경우 sample size와 power 계산 또한 원래의 계획(adjusted, unadjusted)에 맞춰 진행하여야 하고, covariate와 함께 randomization을 진행 할 수 있습니다. 다만, covariate의 개수가 samplesize에 비해 많은 경우 이는 모델이 overfitting되어 예측력이 떨어질 수 있습니다.</p>
</section><section id="사용할-변수에-대해-기저와-함께-추후의-결과-값을-함께-기록해야합니다." class="level3"><h3 class="anchored" data-anchor-id="사용할-변수에-대해-기저와-함께-추후의-결과-값을-함께-기록해야합니다.">사용할 변수에 대해 기저와 함께 추후의 결과 값을 함께 기록해야합니다.</h3>
<p>절대적인 변화값만을 사용하였을때는 그 값의 변화 여부가 정확하게 반영이 되지 않을 수 있습니다.(ex. 혈압이 130에서 150으로 오른 것과 170에서 190으로 오른 것이 임상적으로 시사하는 의미가 다를 수 있습니다) 따라서 기저와 함께 추후의 결과값을 함께 기록하거나, 변화량 대신 기저 측정값을 공변량으로 넣는 것을 권고합니다. 아래의 예시를 통해 살펴보겠습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">150</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">final</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span>, treatment <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">final</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#baseline을 반영하지 않은 모델</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">final</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#basline을 정확히 반영한 모델</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">final</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#차이의 절댓값만 반영한 모델</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">delta</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">final</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">delta</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  Model    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unadjusted"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Baseline adj."</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Delta model"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Estimate <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est0</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est2</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  SE       <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se0</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se2</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>          Model  Estimate        SE
1    Unadjusted -20.02198 0.4667181
2 Baseline adj. -19.82104 0.3184145
3   Delta model -19.65273 0.4278252</code></pre>
</div>
</div>
<p>결과를 살펴보면 알 수 있듯이 baseline을 함께 반영한 모델이 가장 낮은 표준오차를 보여주고, 아예 보정을 하지 않은 모델이나 차이만을 사용한 모델의 경우 다소 예측력이 떨어진다는 것을 알 수 있습니다. 따라서 다음과 같은 권고안 들이 있습니다.</p>
<ul>
<li><p>결과를 변화값 대신 백분율 변화로 정의하려면(Noninferiority trial에서 군간 평균 차이 대신 ratio 쓰고 싶으면) 사전에 심사부서와 결과 정의및 조정방식을 협의해야함</p></li>
<li><p>비율은 non-collapsibility 문제로 해석이 복잡하고, difference 방식이 해석에 더 직관적이라 FDA에서 가장 권장</p></li>
</ul>
<p>Collapsibility의 경우 아래의 예시를 살펴보면 알 수 있습니다.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
<col style="width: 20%">
</colgroup>
<thead><tr class="header">
<th>Subgroup</th>
<th>% of Population</th>
<th>Success Rate (New drug)</th>
<th>Success Rate (Placebo)</th>
<th>Odds Ratio</th>
</tr></thead>
<tbody>
<tr class="odd">
<td>Biomarker‑positive</td>
<td>50%</td>
<td>80.0%</td>
<td>33.3%</td>
<td>8.0</td>
</tr>
<tr class="even">
<td>Biomarker‑negative</td>
<td>50%</td>
<td>25.0%</td>
<td>4.0%</td>
<td>8.0</td>
</tr>
<tr class="odd">
<td><strong>Combined</strong></td>
<td><strong>100%</strong></td>
<td><strong>52.5%</strong></td>
<td><strong>18.7%</strong></td>
<td><strong>4.8</strong></td>
</tr>
</tbody>
</table>
<p>두 그룹에서의 OR이 모두 동일한데, 두 그룹을 합친 경우 OR이 오히려 낮아진다는 것을 알 수 있습니다. 이를 non-collapsibility라 하고, 따라서 collapsible한 변수인 Risk difference와 같은 변수들이 임상적 해석이 더 용이하여 권고한다 되어있습니다.</p>
</section><section id="linear-models" class="level2"><h2 class="anchored" data-anchor-id="linear-models">Linear Models</h2>
<section id="ate평균-효과-추정" class="level3"><h3 class="anchored" data-anchor-id="ate평균-효과-추정">ATE(평균 효과) 추정</h3>
<p>RCT의 경우 선형 모델이 정확하지 않더라고 전체 집단의 평균 효과를 추정하는데는 편향이 없습니다. 따라서 회귀 계수가 전체 모집단 수준에서 추정이 가능하나, 모형이 현실을 더 잘 포착하면 분산이 줄어들어 검정력은 올라가게 됩니다. 아래의 예시를 통해 살펴보겠습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">500</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_unadj</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_unadj</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_lin</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_adj_lin</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_cub</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_adj_cub</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">reps</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tr</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 실제로 y는 tr과 x의 세제곱에 의해서 결정된다고 가정</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tr</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># y와 tr만 살펴봅니다</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tr</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">coefficients</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># y와 tr, 그리고 x의 선형관계만 살펴봅니다</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tr</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">coefficients</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># y와 tr, 그리고 x의 비선형관계를 3차항까지 함께 살펴봅니다</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tr</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/AsIs.html">I</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/AsIs.html">I</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">coefficients</span></span>
<span>  </span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_unadj</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tr"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_unadj</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tr"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_lin</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tr"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_adj_lin</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tr"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_cub</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tr"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Estimate"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_adj_cub</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sm2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tr"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://dplyr.tidyverse.org">dplyr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">tibble</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  Model         <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unadjusted"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Linear adj."</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Cubic adj."</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Mean_Est      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_unadj</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,   <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_lin</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,   <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_cub</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Empirical_SE  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/sd.html">sd</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_unadj</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,     <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/sd.html">sd</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_lin</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,     <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/sd.html">sd</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ests_adj_cub</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Avg_Model_SE  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_unadj</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,     <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_adj_lin</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,     <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_adj_cub</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 3 × 4
  Model       Mean_Est Empirical_SE Avg_Model_SE
  &lt;chr&gt;          &lt;dbl&gt;        &lt;dbl&gt;        &lt;dbl&gt;
1 Unadjusted      2.09        0.954        1.04 
2 Linear adj.     2.00        0.708        0.663
3 Cubic adj.      2.02        0.166        0.180</code></pre>
</div>
</div>
<p>결과를 살펴보게 되면 세 모델 모두 tr의 효과인 2는 잘 포착하였지만, 표준오차의 경우 x의 3차항까지 정확하게 포착한 모델이 가장 정밀하게 포착함을 알 수 있습니다 (Empirical SE와 Avg_SE의 차이가 적을 수록 좋은 모델)</p>
</section><section id="cluster와-이분산성의-보정" class="level3"><h3 class="anchored" data-anchor-id="cluster와-이분산성의-보정">Cluster와 이분산성의 보정</h3>
<p>표준 오차의 경우 default를 1대1 배정 RCT에서는 모델이 misspecify되어도 허용하지만 그렇지 않은 경우엔 sandwich, bootstrap, cluster-robust를 권장한다고 되어있습니다. 아래의 예시를 통해 살펴보겠습니다</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://sandwich.R-Forge.R-project.org/">sandwich</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  </span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lmtest</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>     </span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>      </span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_cl</span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 병원 수</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 병원당 환자 수</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_cl</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 병원 클러스터</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 병원별로 치료군과 비치료군을 반반씩 배정</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, times <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_cl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  levels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"0"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_cl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 병원별 랜덤 효과</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster_eff</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_cl</span>, sd <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사람별로 이분산성이 존재한다고 가정</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sigma</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">eps</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_cl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span>, mean <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, sd <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sigma</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 2) 결과 변수 y 생성</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># y = 1.5*treatment + 0.8*x + 병원효과 + 이분산 오차</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.5</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/character.html">as.character</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>     <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.8</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>     <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster_eff</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>     <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">eps</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 3) OLS 모형 적합</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4.1 Nominal (기본) SE</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_nominal</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># treatment1 계수</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_nominal</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/summary.html">summary</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Std. Error"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4.2 Huber-White sandwich SE</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_sandwich</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">sqrt</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/diag.html">diag</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vcovHC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span>, type <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"HC0"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4.3 병원별 클러스터-로버스트 SE</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_cluster</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">sqrt</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/diag.html">diag</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">vcovCL</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span>,</span>
<span>                                cluster <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster</span>,</span>
<span>                                type    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"HC0"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4.4 부트스트랩 (개별 환자 단위 리샘플링)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_ind</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/boot/man/boot.html">boot</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>, statistic <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_ind</span>, R <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_boot_ind</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">t</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> </span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_boot_ind</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/sd.html">sd</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">t</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>    </span>
<span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_clust</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">clusters</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sel</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/unique.html">unique</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">clusters</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>                <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/unique.html">unique</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">clusters</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>                replace <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d2</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,</span>
<span>                <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sel</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">clusters</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cl</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_coefs</span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">replicate</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">boot_clust</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cluster</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_boot_clust</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_coefs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_boot_clust</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/sd.html">sd</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_coefs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  Method       <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Nominal"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sandwich"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Cluster-robust"</span>,</span>
<span>                   <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Bootstrap(indiv.)"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Bootstrap(cluster)"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  Estimate     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_nominal</span>,  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_nominal</span>,  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_nominal</span>,</span>
<span>                   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_boot_ind</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">est_boot_clust</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  StdError     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_nominal</span>,   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_sandwich</span>,   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_cluster</span>,</span>
<span>                   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_boot_ind</span>,  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_boot_clust</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/print.html">print</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>              Method Estimate   StdError
1            Nominal 1.476932 0.13445377
2           Sandwich 1.476932 0.13428108
3     Cluster-robust 1.476932 0.06100908
4  Bootstrap(indiv.) 1.492976 0.14092968
5 Bootstrap(cluster) 1.480728 0.05974858</code></pre>
</div>
</div>
<p>결과를 살펴보면 모든 모델이 실제 치료효과인 1.5는 정확하게 포착한 것을 알 수 있습니다. 다만, cluster와 이분산성이 존재하는 경우 sandwich estimator의 경우 이분산을 보정하여 nominal보다 조금 더 나은 오차를 그리고 cluster-robust와 cluster를 반영한 bootstrapping의 경우 더 정밀한 표준오차를 제공하는 것을 알 수 있습니다. 따라서 cluster와 이분산성이 존재할 수 있는 데이터의 경우 이를 고려한 통계분석을 하는 것이 결과가 보다 정확하다는 것을 알 수 있습니니다.</p>
</section><section id="치료효과와-공변량-사이의-상호작용" class="level3"><h3 class="anchored" data-anchor-id="치료효과와-공변량-사이의-상호작용">치료효과와 공변량 사이의 상호작용</h3>
<p>선형모델에서 treatment x covariate를 넣을 수 있지만 ATE의 경우 평범한 선형모델의 추정량으로도 사용이 가능합니니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10000</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">biomarker</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Uniform.html">runif</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료결과는 치료변수와 biomarker와의 상호작용에도 영향을 받는다고 가정</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">biomarker</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">biomarker</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>, treatment<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">biomarker</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">biomarker</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 조건부 효과(conditional effect) 계산</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">beta1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">beta3</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1:biomarker"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">effects_subgroups</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  biomarker <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">50</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  effect     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">beta1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">beta3</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">50</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 평균처리효과(ATE) 계산</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mean_bio</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">biomarker</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ate_hat</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">beta1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">beta3</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mean_bio</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vc</span>       <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/vcov.html">vcov</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_ate</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">sqrt</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mean_bio</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">^</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1:biomarker"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1:biomarker"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span></span>
<span>  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mean_bio</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"treatment1:biomarker"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  conditional_effects <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">effects_subgroups</span>,</span>
<span>  ATE                 <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ate_hat</span>,</span>
<span>  SE_ATE              <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_ate</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>$conditional_effects
  biomarker   effect
1        10 5.808792
2        50 7.693326
3        90 9.577859

$ATE
treatment1 
   7.68178 

$SE_ATE
[1] 0.2003858</code></pre>
</div>
</div>
<p>결과를 살펴보면 biomarker의 값에 따라 치료효과가 달라진다는 것을 알 수 있습니다. biomarker(0-100사이의 정규분포)가 50일때의 치료효과가 기본적인 모델로 했을때의 값인 7.6과 가장 유사함을 알 수 있습니다. 따라서 두 모델을 모두 보여주는 것이 이상적이나 전체 치료효과를 보여주는데는 기본적인 모델을 써도 기초적인 기술은 가능하다는 것을 알 수 있습니다.</p>
</section></section><section id="nonlinear-model" class="level2"><h2 class="anchored" data-anchor-id="nonlinear-model">Nonlinear model</h2>
<p>모든 치료 결과 변수들이 연속변수일 수 없기 때문에 비선형 모델을 적합해야하는 경우가 많습니다.(생존분석, 결과가 이진변수) 기본적인 권고사항은 선형모델을 적합할때와 유사합니다. Subgroup 별로 treatment effect가 다를 수 있는데, OR HR 은 아까 얘기한 non-collapsibility가 존재하기 때문에 collapsible한 변수를 권고한다는 내용이 있습니다. 또한, 추정량을 사전에 정의할떄 conditional인지 unconditional인지 분명히 해야합니다.</p>
<section id="층화된-모델에서-binary-결과의-분석" class="level3"><h3 class="anchored" data-anchor-id="층화된-모델에서-binary-결과의-분석">층화된 모델에서 binary 결과의 분석</h3>
<p>Cochran-Mantel-Hasenszel(CMH) 방법은 binary 결과이고 covariate로 층화된 상태에서 조건부 치료 효과를 추정할떄 사용됩니다. 각 층마다 OR이 동일하다는 가정하고 층화된 카이제곱 검정 같은 분석이라고 보시면 됩니다. 예시 결과는 아래와 같습니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_strata</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">OR_true</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">stratum</span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_strata</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, times <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_strata</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">base_p</span>     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.2</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.3</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, each <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n_per</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">logit</span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Log.html">log</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">expit</span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Log.html">exp</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_control</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">base_p</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_treated</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">expit</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/boot/man/logit.html">logit</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">base_p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Log.html">log</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">OR_true</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">stratum</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span>            <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_treated</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_control</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">stratum</span>,</span>
<span>                 treatment <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>, levels<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>                 <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tab</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/with.html">with</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/table.html">table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">stratum</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treatment</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/mantelhaen.test.html">mantelhaen.test</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">tab</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>
    Cochran-Mantel-Haenszel test

data:  tab
Cochran-Mantel-Haenszel M^2 = 3.2922, df = 3, p-value = 0.3487</code></pre>
</div>
</div>
<p>결과를 살펴보시면 p-value가 0.05이상으로 치료여부에 따른 결과의 차이가 없다는 것을 알 수 있습니다. 다만 CMH 방법의 경우 범주형 변수의 경우 사용이 가능하고 연속형 변수의 경우 로지스틱 회귀 분석을 사용할 수 있습니다. 비선형 모형을 쓸 때는misspecify 되지 않게 분석상의 함의를 담당 심사부서와 논의해야 한다는 내용또한 포함되어 있습니다. 즉, 선형모델에서 설명드린 바와 같이 공변량을 조정할때는 타당한 통계가정과 적합한 통계방법을 통해 표준오차를 토출해야합니다. 문서에서 제시한 예시방법(g-computation)과 reliable하다고 언급한 방법인 IPTW 분석을 살펴보겠습니다.</p>
<section id="문서에서-제시한-예시방법g-computation" class="level4"><h4 class="anchored" data-anchor-id="문서에서-제시한-예시방법g-computation">문서에서 제시한 예시방법(g-computation)</h4>
<ul>
<li><p>치료(treatment)와 사전 지정된 기저 공변량을 설명변수로 하는 로지스틱 회귀모형을 최우추정법(MLE)으로 적합한다.</p></li>
<li><p>각 피험자마다, 치료군으로 가정했을 때의 반응 확률을 기저 공변량으로 예측한다.</p></li>
<li><p>2단계에서 계산된 모든 피험자의 예측 확률을 평균내어 “치료 시 평균 반응 확률”을 구한다.</p></li>
<li><p>각 피험자마다, 대조군으로 가정했을 때의 반응 확률을 기저 공변량으로 예측한다.</p></li>
<li><p>4단계 예측 확률의 평균을 내어 “대조 시 평균 반응 확률”을 구한다.</p></li>
<li><p>이 두 평균 반응률을 바탕으로 위험차(risk difference), 상대위험(relative risk), 오즈비(odds ratio) 등 원하는 <strong>무조건적 처리 효과(unconditional treatment effect)</strong>를 계산한다.</p></li>
</ul></section><section id="예시" class="level4"><h4 class="anchored" data-anchor-id="예시">예시</h4>
<ul>
<li>고혈압 환자 100명을 대상으로, 1년 후 뇌졸중 발생 여부를 평가 RCT</li>
<li>표준 vs 신약</li>
<li>Covariate: age, baseline_SBP</li>
<li>Result: Stroke</li>
<li>신약의 효과(RR, RD, OR)을 g-computation으로 추정</li>
</ul></section><section id="g-computation" class="level4"><h4 class="anchored" data-anchor-id="g-computation">g computation</h4>
<p>g computation의 경우 로지스틱 회귀분석으로 개인별로 치료군과 혈압에 따라 뇌졸중이 발생할 확률을 계산합니다. 이후 모든 환자를 신약군이라 가정하여 기저 혈압으로 부터 뇌졸중이 발생할 확률을 평균화하여 구하고, 이후 모든 환자가 대조군일떄를 가정하여 기저 혈압으로부터 뇌졸중이 발생할 확률을 구하고 차이를 계산합니다.</p>
</section><section id="iptw" class="level4"><h4 class="anchored" data-anchor-id="iptw">IPTW</h4>
<p>IPTW의 경우 각 환자가 혈압에 따라 신약군에 포함될 확률을 계산한 이후, 그 확률의 역수를 가중치로 주어 공변량을 보정하는 방법입니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://dplyr.tidyverse.org">dplyr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>                             <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># RCT 대상자 수</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline_SBP</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">rnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">150</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 기저 혈압</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat_num</span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 무작위배정</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span>        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat_num</span>, levels<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lp0</span>          <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline_SBP</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">150</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lp1</span>          <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lp0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1.5</span> </span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p0</span>           <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Logistic.html">plogis</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lp0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p1</span>           <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Logistic.html">plogis</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lp1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>            <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Binomial.html">rbinom</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat_num</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p1</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline_SBP</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">calc_measures</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Unadjusted</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_un</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_un</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"0"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rd_un</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_un</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_un</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># g-computation</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m_gc</span>   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline_SBP</span>, family<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">binomial</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d1</span>     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/transform.html">transform</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span>, treat<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,levels<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d0</span>     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/transform.html">transform</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span>, treat<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,levels<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_gc</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m_gc</span>,newdata<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d1</span>,type<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"response"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_gc</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/mean.html">mean</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">m_gc</span>,newdata<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d0</span>,type<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"response"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rd_gc</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_gc</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_gc</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rr_gc</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_gc</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_gc</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">or_gc</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_gc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_gc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_gc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_gc</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># IPTW</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ps_mod</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">as.numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/character.html">as.character</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">baseline_SBP</span>,</span>
<span>                family<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">binomial</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ps</span>     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ps_mod</span>,type<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"response"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">w</span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1"</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ps</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ps</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_ip</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sum.html">sum</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">w</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sum.html">sum</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">w</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_ip</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sum.html">sum</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">w</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">y</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"0"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sum.html">sum</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">w</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">d</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">treat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"0"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rd_ip</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_ip</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_ip</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rr_ip</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_ip</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_ip</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">or_ip</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_ip</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_ip</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_ip</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r0_ip</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rd_un</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_gc</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">r1_ip</span>,</span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rd_gc</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rr_gc</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">or_gc</span>,</span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rd_ip</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rr_ip</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">or_ip</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 3) 원본 지표 계산</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">calc_measures</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4) 부트스트랩으로 SE 계산 (B=200)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">B</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_vals</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">replicate</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">B</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">idx</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">n</span>, replace<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">calc_measures</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">df</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">idx</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/apply.html">apply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_vals</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sd</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 5) 결과 정리</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  Method      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Unadjusted"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"G-computation"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"IPTW"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  RiskDiff    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  SE_RiskDiff <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  RR          <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>,       <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  SE_RR       <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>,       <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  OR          <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>,       <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">orig</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">9</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  SE_OR       <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>,       <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">se_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">9</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/print.html">print</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">res</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>         Method   RiskDiff SE_RiskDiff        RR     SE_RR        OR     SE_OR
1    Unadjusted -0.1041667  0.07519916        NA        NA        NA        NA
2 G-computation -0.1567208  0.05039919 0.4074673 0.1410154 0.3358952 0.1375932
3          IPTW -0.1386437  0.06140822 0.4791533 0.1800494 0.4030100 0.1910830</code></pre>
</div>
</div>
<p>결과를 살펴보면 표준오차가 줄어 정밀도가 개선되었다는 것을 알 수 있습니다.(실제 RD는 -0.136) 이는 G-computation과 IPTW 모두 공변량이 random imbalance가 발생하는 경우 이를 가중치 혹은 확률 계산을 통해 조정할 뿐만 아니라 이 과정에서 공변량이 모델에 반영되어 잔차분산도 줄여주기 때문입니다. 특히 n수가 적어 random imbalance가 발생하기 쉬운 경우 더 값을 정확하게 추정할 수 있습니다.</p>


</section></section></section></section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{jo2025,
  author = {Jo, Hyungwoo},
  title = {Fda의 {RCT} {Guidance} for Industry},
  date = {2025-04-18},
  url = {https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-jo2025" class="csl-entry quarto-appendix-citeas">
Jo, Hyungwoo. 2025. <span>“Fda의 RCT Guidance for Industry.”</span>
April 18, 2025. <a href="https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/">https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/</a>.
</div></div></section></div> ]]></description>
  <category>presentation</category>
  <guid>https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/</guid>
  <pubDate>Fri, 18 Apr 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-04-18-FDA_RCT/img/logo.png" medium="image" type="image/png" height="89" width="144"/>
</item>
<item>
  <title>ADaM Compliant ADSL Dataset Generation</title>
  <dc:creator>Hojun LEE</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-04-08-ADaM/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="objectives" class="level1"><h1>OBJECTIVES</h1>
<ul>
<li>CDISC: SDTM과 ADaM이 무엇인지 알아보자.</li>
<li>직접 SDTM을 ADaM으로 변환해보자.</li>
<li>ADaM을 이용, TLG(table, list, graphics)를 빠르게 만들어보자.</li>
<li>Shiny module을 이용, ADaM dataset에서 복잡한 분석을 빠르게 수행해보자.</li>
</ul></section><section id="시작하기-전에" class="level1"><h1>시작하기 전에</h1>
<p>이전에 CDISC와 CDISC 中 ADaM, ADSL에 간단히 다룬 적이 있다. <br><a href="https://blog.zarathu.com/posts/2023-04-28-tidycdisc/"><code>zarathu_CDISC</code></a> <br> 본문을 읽기 전, 먼저 위 문서를 읽고 오는 것을 추천한다.</p>
</section><section id="cdisc-sdtm-and-adam" class="level1"><h1>CDISC: SDTM and ADaM</h1>
<p><strong>1. CDISC 표준 개요</strong></p>
<ul>
<li>
<strong>목적</strong>: 임상시험 데이터의 <strong>표준화</strong> 및 <strong>재현성</strong> 확보</li>
<li>
<strong>주요 구성</strong>:
<ul>
<li>
<code>SDTM</code> (원시 데이터)</li>
<li>
<code>ADaM</code> (분석용 데이터)</li>
<li>
<code>Define-XML</code> (메타데이터)</li>
</ul>
</li>
</ul>
<p><strong>2. SDTM (Study Data Tabulation Model)</strong></p>
<ul>
<li>
<strong>역할</strong>: 원시 데이터 수집 표준</li>
<li>
<strong>특징</strong>:
<ul>
<li>관측 중심 구조 (1행 = 1관측치)</li>
<li>도메인 별 분류 (DM, VS, LB 등)</li>
<li>예시: <code>VS</code>(Vital Signs), <code>LB</code>(Lab Data)</li>
</ul>
</li>
</ul>
<p><strong>3. ADaM (Analysis Data Model)</strong></p>
<ul>
<li>
<strong>등장 이유</strong>: SDTM의 분석 한계 보완
<ul>
<li>분석용 파생변수 부재</li>
<li>치료군 비교 어려움</li>
<li>재현성 문제</li>
</ul>
</li>
<li>
<strong>핵심 특징</strong>:
<ul>
<li>분석 친화적 구조</li>
<li>파생변수 명시적 정의</li>
<li>데이터 트레일(traceability) 강화</li>
</ul>
</li>
</ul>
<p><strong>4. 주요 ADaM 데이터셋</strong></p>
<table class="caption-top table">
<thead><tr class="header">
<th>데이터셋</th>
<th>용도</th>
<th>예시 변수</th>
</tr></thead>
<tbody>
<tr class="odd">
<td><strong>ADSL</strong></td>
<td>기본 분석</td>
<td>
<code>TRT01P</code>, <code>ITTFL</code>
</td>
</tr>
<tr class="even">
<td><strong>ADAE</strong></td>
<td>이상반응</td>
<td>
<code>TRTEMFL</code>, <code>AOCCFL</code>
</td>
</tr>
<tr class="odd">
<td><strong>ADPC</strong></td>
<td>약동학</td>
<td>
<code>DV</code>, <code>LLOQ</code>
</td>
</tr>
<tr class="even">
<td><strong>ADTTE</strong></td>
<td>생존분석</td>
<td>
<code>AVAL</code>, <code>CNSR</code>
</td>
</tr>
<tr class="odd">
<td><strong>ADLB</strong></td>
<td>검사실</td>
<td>
<code>ANRIND</code>, <code>BASE</code>
</td>
</tr>
</tbody>
</table>
<p><strong>5. 표준 적용 흐름</strong></p>
<div class="cell" data-layout-align="default">
<div class="cell-output-display">
<div>
<p></p><figure class="figure"></figure><p></p>
<div>
<pre class="mermaid mermaid-js">flowchart LR
    원시데이터["원시 데이터"] --&gt;|SDTM 변환| DM[DM domain]
    DM --&gt;|ADaM 처리| ADSL[ADSL dataset]
    ADSL --&gt;|통계 분석| 리포트["통계 보고서"]
</pre>
</div>
<p></p>
</div>
</div>
</div>
<section id="sdtm-vs-adam-변수-비교" class="level2"><h2 class="anchored" data-anchor-id="sdtm-vs-adam-변수-비교">SDTM vs ADaM 변수 비교</h2>
<p><strong>1. SDTM 주요 변수 구조</strong></p>
<table class="caption-top table">
<colgroup>
<col style="width: 32%">
<col style="width: 44%">
<col style="width: 24%">
</colgroup>
<thead><tr class="header">
<th>도메인</th>
<th>핵심 변수</th>
<th>설명</th>
</tr></thead>
<tbody>
<tr class="odd">
<td><strong>DM</strong></td>
<td>
<code>USUBJID</code>, <code>AGE</code>, <code>SEX</code>, <code>RACE</code>, <code>ARM</code>
</td>
<td>인구통계학 및 치료군 정보</td>
</tr>
<tr class="even">
<td><strong>AE</strong></td>
<td>
<code>AETERM</code>, <code>AESTDTC</code>, <code>AEENDTC</code>, <code>AESEV</code>
</td>
<td>이상반응 데이터</td>
</tr>
<tr class="odd">
<td><strong>VS</strong></td>
<td>
<code>VSTEST</code>, <code>VSORRES</code>, <code>VSSTRESN</code>
</td>
<td>생체징후 데이터</td>
</tr>
<tr class="even">
<td><strong>LB</strong></td>
<td>
<code>LBTEST</code>, <code>LBORRES</code>, <code>LBSTRESN</code>
</td>
<td>검사실 데이터</td>
</tr>
<tr class="odd">
<td><strong>EX</strong></td>
<td>
<code>EXTRT</code>, <code>EXDOSE</code>, <code>EXSTDTC</code>
</td>
<td>투여 정보</td>
</tr>
</tbody>
</table>
<p><strong>2. ADaM으로 전환 시 주요 변경 사항</strong></p>
<table class="caption-top table">
<thead><tr class="header">
<th>SDTM 변수</th>
<th>ADaM 변환</th>
<th>변경 이유</th>
</tr></thead>
<tbody>
<tr class="odd">
<td>
<code>AESTDTC</code>(문자형)</td>
<td>
<code>ASTDT</code>(숫자형 날짜)</td>
<td>분석용 날짜 포맷</td>
</tr>
<tr class="even">
<td><code>AGE</code></td>
<td>
<code>AGEGR1</code>(연령그룹)</td>
<td>카테고리 분석</td>
</tr>
<tr class="odd">
<td><code>ARM</code></td>
<td>
<code>TRT01P</code>(계획 치료군)</td>
<td>프로토콜 위반 처리</td>
</tr>
<tr class="even">
<td>-</td>
<td>
<code>TRTEMFL</code>(치료기간 발생 여부)</td>
<td>안전성 분석 필터링</td>
</tr>
</tbody>
</table>
<p><strong>ADaM 전용 파생 변수</strong></p>
<ol type="1">
<li>
<strong>기본 변수</strong>:
<ul>
<li>
<code>AVAL</code>: 분석용 수치값 (예: LB도메인의 <code>VSSTRESN</code> → <code>AVAL</code>)</li>
<li>
<code>CHG</code>: 기준값 대비 변화량</li>
<li>
<code>PCHG</code>: 백분율 변화량</li>
</ul>
</li>
<li>
<strong>플래그 변수</strong>:
<ul>
<li>
<code>SAFFL</code>: 안전성 평가 집단 플래그</li>
<li>
<code>ITTFL</code>: Intent-to-Treat 플래그</li>
<li>
<code>ANL01FL</code>: 주요 분석 플래그</li>
</ul>
</li>
<li>
<strong>날짜 변수</strong>:
<ul>
<li>
<code>ADT</code>: 분석용 숫자형 날짜</li>
<li>
<code>ADY</code>: 기준일(DAY 1) 대비 일수</li>
</ul>
</li>
</ol>
<div class="cell" data-layout-align="default">
<div class="cell-output-display">
<div>
<p></p><figure class="figure"></figure><p></p>
<div>
<pre class="mermaid mermaid-js">flowchart TB
    SDTM["SDTM Domain: LB"] --&gt;|Transform| ADAM["ADaM ADLB Dataset"]
    
    SDTM --&gt; LB1[("LBTEST(Test Name)")]
    SDTM --&gt; LB2[("LBORRES(Result)")]
    SDTM --&gt; LB3[("LBDTC(Date)")]
    
    ADAM --&gt; AD1[("PARAMCD(Standard Code)")]
    ADAM --&gt; AD2[("AVAL(Numeric Value)")]
    ADAM --&gt; AD3[("ADT(Analysis Date)")]
    ADAM --&gt; AD4[("ANRIND(Reference Range)")]
    
    ADAM --&gt; Analysis[["Statistical Analysis"]]
</pre>
</div>
<p></p>
</div>
</div>
</div>
</section></section><section id="sdtm-to-adam" class="level1"><h1>SDTM to ADaM</h1>
<p>SDTM에서 ADaM으로 바뀌는 과정을 돕기 위해 <code>pharmaverse</code>에서 패키지를 만들었다. 이를 이용하여 직접 변환을 해보자.</p>
<p>오늘 다룰 대표적인 ADaM은 아래와 같다.</p>
<ul>
<li>ADSL(Analysis Dataset, subject-level)</li>
<li>ADAE(Analysis Dataset, Adverse Effect)</li>
</ul>
<p>위 ADaM들을 pharmaverse packages를 이용해 다뤄보자.</p>
<section id="adsl" class="level2"><h2 class="anchored" data-anchor-id="adsl">ADSL</h2>
<p>ADSL은 임상 시험에서 환자별 기초 정보를 담는 핵심 데이터셋으로, 주요 변수와 분석에 활용된다.</p>
<p><strong>목적</strong>:<br> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 임상 시험 대상자(Subject)의 인구통계학적 정보, 치료 그룹, 기초 값 등 개인 수준 데이터를 제공. <br><strong>구조</strong>:<br> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 한 행에 한 환자의 정보가 포함되며, <code>USUBJID</code> (고유 환자 식별자)로 구분. <br><strong>활용</strong>:<br> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 효능/안전성 분석의 기초 데이터로 사용. <br></p>
<p><strong>주요 변수 (Columns)</strong></p>
<table class="caption-top table">
<colgroup>
<col style="width: 23%">
<col style="width: 30%">
<col style="width: 46%">
</colgroup>
<thead><tr class="header">
<th>Category</th>
<th>Variables</th>
<th>Description</th>
</tr></thead>
<tbody>
<tr class="odd">
<td><strong>Identifiers</strong></td>
<td>• <code>USUBJID</code><br>• <code>SUBJID</code><br>• <code>SITEID</code>
</td>
<td>• 고유 환자 ID<br>• 대상자 ID<br>• 연구기관 ID</td>
</tr>
<tr class="even">
<td><strong>Demographics</strong></td>
<td>• <code>AGE</code><br>• <code>SEX</code><br>• <code>RACE</code><br>• <code>COUNTRY</code>
</td>
<td>• 연령<br>• 성별<br>• 인종<br>• 국가</td>
</tr>
<tr class="odd">
<td><strong>Treatment</strong></td>
<td>• <code>TRT01P</code><br>• <code>TRT01A</code><br>• <code>TRTGRPn</code>
</td>
<td>• 계획 치료<br>• 실제 치료<br>• 치료그룹</td>
</tr>
<tr class="even">
<td><strong>Trial</strong></td>
<td>• <code>ARM</code><br>• <code>ACTARM</code><br>• <code>RFSTDTC</code><br>• <code>RFENDTC</code>
</td>
<td>• 할당 치료군<br>• 실제 치료군<br>• 첫 투여일<br>• 마지막 방문일</td>
</tr>
<tr class="odd">
<td><strong>Flags</strong></td>
<td>• <code>SAFFL</code><br>• <code>ITTFL</code><br>• <code>COMPLFL</code>
</td>
<td>• 안전성 평가 대상<br>• ITT 대상<br>• 연구 완료 여부</td>
</tr>
<tr class="even">
<td><strong>CDISC</strong></td>
<td>All variables</td>
<td>CDISC ADaM 표준 준수</td>
</tr>
</tbody>
</table>
<p><br></p>
<section id="adsl_example" class="level3"><h3 class="anchored" data-anchor-id="adsl_example">ADSL_example</h3>
<p>예제는 아래와 같은 순서로 진행한다.</p>
<ul>
<li>data/Packages loading</li>
<li>Derivation Building</li>
<li>Variables Grouping</li>
<li>Derive exposure variables</li>
<li>Derive Treatment Variables</li>
<li>Derive Disposition Variables</li>
<li>Derive Cause of Death</li>
<li>Derive Other Grouping Variables</li>
<li>Applying metadate</li>
</ul>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#load packages</span></span>
<span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://atorus-research.github.io/metacore/">metacore</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/pharmaverse/metatools">metatools</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://pharmaverse.github.io/pharmaversesdtm/">pharmaversesdtm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://pharmaverse.github.io/admiral/">admiral</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://atorus-research.github.io/xportr/">xportr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://dplyr.tidyverse.org">dplyr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://tidyr.tidyverse.org">tidyr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://lubridate.tidyverse.org">lubridate</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://stringr.tidyverse.org">stringr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/rstudio/DT">DT</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Read in input SDTM data</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">suppdm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">suppdm</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#in case; importing SAS datasets using haven::read_sas()</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># NA가 아닌 ""로 읽히기 때문에, NA처리를 따로 해줘야한다.</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">suppdm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">suppdm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>이후 편의를 위해 dm과 suppdm을 아래와 같이 합칠 수 있다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/names.html">names</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  [1] "STUDYID"  "DOMAIN"   "USUBJID"  "SUBJID"   "RFSTDTC"  "RFENDTC"  "RFXSTDTC" "RFXENDTC" "RFICDTC"  "RFPENDTC" "DTHDTC"   "DTHFL"    "SITEID"  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [14] "AGE"      "AGEU"     "SEX"      "RACE"     "ETHNIC"   "ARMCD"    "ARM"      "ACTARMCD" "ACTARM"   "COUNTRY"  "DMDTC"    "DMDY"    </span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/names.html">names</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">suppdm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [1] "STUDYID"  "RDOMAIN"  "USUBJID"  "IDVAR"    "IDVARVAL" "QNAM"     "QLABEL"   "QVAL"     "QORIG"    "QEVAL"   </span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># merge 대신 suppdm을 통해 간편하게 확인할 수 있다.</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm_suppdm</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">combine_supp</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">suppdm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/names.html">names</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm_suppdm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#  [1] "STUDYID"  "DOMAIN"   "USUBJID"  "SUBJID"   "RFSTDTC"  "RFENDTC"  "RFXSTDTC" "RFXENDTC" "RFICDTC"  "RFPENDTC" "DTHDTC"   "DTHFL"    "SITEID"  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [14] "AGE"      "AGEU"     "SEX"      "RACE"     "ETHNIC"   "ARMCD"    "ARM"      "ACTARMCD" "ACTARM"   "COUNTRY"  "DMDTC"    "DMDY"     "IDVARVAL"</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># [27] "COMPLT16" "COMPLT24" "COMPLT8"  "EFFICACY" "ITT"      "SAFETY"</span></span></code></pre></div></div>
</div>
<p>그 이후 specification file을 metacore object로 합친다. <br><code>metacore::spec_to_metacore()</code> <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Read in metacore object</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">spec_to_metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  path <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"./safety_specs.xlsx"</span>,</span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># All datasets are described in the same sheet</span></span>
<span>  where_sep_sheet <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select_dataset</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADSL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><strong>Derivation Building</strong></p>
<p>이제 SDTM dataset(DM, SUPPDM)에서 필요한 변수들을 추출하여 ADaM(ADSL)을 생성하는 과정이 시작된다. <code>metatools:build_from_derived()</code></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_preds</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">build_from_derived</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>,</span>
<span>  ds_list <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"dm"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm_suppdm</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"suppdm"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dm_suppdm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>  predecessor_only <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, keep <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stderr">
<pre><code>Not all datasets provided. Only variables from DM, SUPPDM will be gathered.</code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_preds</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-5b9146acb9fb3a6c9a02" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-5b9146acb9fb3a6c9a02">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034"],["1015","1023","1028","1033","1034"],["701","701","701","701","701"],["USA","USA","USA","USA","USA"],[63,64,71,74,77],["YEARS","YEARS","YEARS","YEARS","YEARS"],["F","M","M","M","F"],["WHITE","WHITE","WHITE","WHITE","WHITE"],["HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO"],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose"],["Pbo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi"],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose"],["Pbo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi"],[null,null,null,null,null],[null,null,null,null,null],["Y","Y","Y","Y","Y"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>COUNTRY<\/th>\n      <th>AGE<\/th>\n      <th>AGEU<\/th>\n      <th>SEX<\/th>\n      <th>RACE<\/th>\n      <th>ETHNIC<\/th>\n      <th>ARM<\/th>\n      <th>ARMCD<\/th>\n      <th>ACTARM<\/th>\n      <th>ACTARMCD<\/th>\n      <th>DTHDTC<\/th>\n      <th>DTHFL<\/th>\n      <th>ITTFL<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":6},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"COUNTRY","targets":5},{"name":"AGE","targets":6},{"name":"AGEU","targets":7},{"name":"SEX","targets":8},{"name":"RACE","targets":9},{"name":"ETHNIC","targets":10},{"name":"ARM","targets":11},{"name":"ARMCD","targets":12},{"name":"ACTARM","targets":13},{"name":"ACTARMCD","targets":14},{"name":"DTHDTC","targets":15},{"name":"DTHFL","targets":16},{"name":"ITTFL","targets":17}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br></p>
<p><strong>Variable Grouping</strong></p>
<p>이제 subgroup을 만들어보자. <br><code>admiral::derive_vars_cat()</code></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">agegr1_lookup</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">condition</span>,            <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGEGR1</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGEGR1N</span>,</span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,          <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Missing"</span>,        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>,</span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span>,                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;18"</span>,        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">between</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"18-64"</span>,        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,</span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,             <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&gt;64"</span>,        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_cat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_cat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  dataset <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_preds</span>,</span>
<span>  definition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">agegr1_lookup</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_cat</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">contains</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ID"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AGE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-8cbb9821a4e54d17ffc1" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-8cbb9821a4e54d17ffc1">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["1015","1023","1028","1033","1034","1047"],["701","701","701","701","701","701"],[63,64,71,74,77,85],["YEARS","YEARS","YEARS","YEARS","YEARS","YEARS"],["18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64"],[2,2,3,3,3,3]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>AGE<\/th>\n      <th>AGEU<\/th>\n      <th>AGEGR1<\/th>\n      <th>AGEGR1N<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[5,8]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"AGE","targets":5},{"name":"AGEU","targets":6},{"name":"AGEGR1","targets":7},{"name":"AGEGR1N","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br> 만일 metacore data(.xlsx)가 terminology definition을 갖고있다면, <br><code>get_control_term()</code>을 이용해 정의를 불러올 수 있다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">get_control_term</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, variable <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGEGR1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code># A tibble: 3 × 2
  code  decode
  &lt;chr&gt; &lt;chr&gt; 
1 &lt;18   &lt;18   
2 18-64 18-64 
3 &gt;64   &gt;64   </code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_ct</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_preds</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">create_cat_var</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>,</span>
<span>    ref_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span>,</span>
<span>    grp_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGEGR1</span>, num_grp_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGEGR1N</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_ct</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">contains</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ID"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AGE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-08558a0c116d4d643e39" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-08558a0c116d4d643e39">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["1015","1023","1028","1033","1034","1047"],["701","701","701","701","701","701"],[63,64,71,74,77,85],["YEARS","YEARS","YEARS","YEARS","YEARS","YEARS"],["18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64"],[2,2,3,3,3,3]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>AGE<\/th>\n      <th>AGEU<\/th>\n      <th>AGEGR1<\/th>\n      <th>AGEGR1N<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[5,8]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"AGE","targets":5},{"name":"AGEU","targets":6},{"name":"AGEGR1","targets":7},{"name":"AGEGR1N","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br> 기존의 방식을 이용해서 진행해도 무방하다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">format_agegr1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">case_when</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;18"</span>,</span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">between</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"18-64"</span>,</span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&gt;64"</span>,</span>
<span>    <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Missing"</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">format_agegr1n</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">case_when</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">between</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,</span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">64</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>,</span>
<span>    <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_cat3</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_preds</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    AGEGR1 <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">format_agegr1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    AGEGR1N <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">format_agegr1n</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AGE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_cat3</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">contains</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ID"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AGE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-f85a04176cb68ed04b83" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-f85a04176cb68ed04b83">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["1015","1023","1028","1033","1034","1047"],["701","701","701","701","701","701"],[63,64,71,74,77,85],["YEARS","YEARS","YEARS","YEARS","YEARS","YEARS"],["18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64"],[2,2,3,3,3,3]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>AGE<\/th>\n      <th>AGEU<\/th>\n      <th>AGEGR1<\/th>\n      <th>AGEGR1N<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[5,8]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"AGE","targets":5},{"name":"AGEU","targets":6},{"name":"AGEGR1","targets":7},{"name":"AGEGR1N","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br> 인종, 성별 등 고정되어 있는 code는 codelist를 참조, 아래와 같이 변경할 수 있다. 다만, codelist에 정의되어 있어야한다.</p>
<p><code>metatools::create_var_from_codelist()</code></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_ct</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_ct</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">create_var_from_codelist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    metacore <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>,</span>
<span>    input_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACE</span>,</span>
<span>    out_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACEN</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_ct</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">contains</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ID"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AGEGR"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RACE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-78f46eb1060f30bf1d71" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-78f46eb1060f30bf1d71">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["1015","1023","1028","1033","1034","1047"],["701","701","701","701","701","701"],["18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64"],[2,2,3,3,3,3],["WHITE","WHITE","WHITE","WHITE","WHITE","WHITE"],[4,4,4,4,4,4]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>AGEGR1<\/th>\n      <th>AGEGR1N<\/th>\n      <th>RACE<\/th>\n      <th>RACEN<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[6,8]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"AGEGR1","targets":5},{"name":"AGEGR1N","targets":6},{"name":"RACE","targets":7},{"name":"RACEN","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<ul>
<li><p><code>create_var_from_codelist</code> 사용: <br> 그룹이 이미 문자형으로 존재하고, 숫자 코드만 필요할 때 <br> (예: <code>RACE</code> → <code>RACEN</code> 변환/ 인종, 성별 등 factor variable)</p></li>
<li><p><code>create_cat_var</code> 사용: <br> 연속형 데이터를 처음부터 그룹핑해야 할 때 <br> (예: <code>AGE</code>(숫자) → <code>AGEGR1</code>(문자) + <code>AGEGR1N</code>(숫자) 생성/ <code>AGE</code>, <code>BMI</code> 등 continuous variable)</p></li>
</ul>
<p><strong>Exposure Derivations</strong></p>
<p>기존 변수들을 바탕으로 새로운 변수들을 생성해보자. 관련 변수들의 예시는 <br><a href="https://pharmaverse.github.io/admiral/articles/adsl.html"><code>admiral</code></a>에서 더 자세한 정보를 확인 가능하다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># DTC &gt; DTM: `derive_vars_dtm`</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex_ext</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dtm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSTDTC</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 시작 일시</span></span>
<span>    new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EXST"</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># EXSTDTM</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dtm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDTC</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 종료 일시 </span></span>
<span>    new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EXEN"</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># EXENDTM</span></span>
<span>    time_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"last"</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#종료 시간 결측 시 "23:59"로 대체.</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 다음과 같은 변수 생성: EXEN, EXSTTMF, EXENDTM, EXENTMF</span></span>
<span>    </span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ADSL에 병합</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_raw</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_ct</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Treatment Start Datetime</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex_ext</span>,</span>
<span>    filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXTRT</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PLACEBO"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSTDTM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>TRTSDTM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSTDTM</span>, TRTSTMF <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSTTMF</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    order <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSTDTM</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSEQ</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 가장 빠른 시작 일시 + 낮은 EXSEQ</span></span>
<span>    mode <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"first"</span>,                  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 첫번째 기록 선택</span></span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Treatment End Datetime</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex_ext</span>,</span>
<span>    filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXTRT</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PLACEBO"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDTM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>TRTEDTM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDTM</span>, TRTETMF <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENTMF</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    order <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDTM</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXSEQ</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 가장 늦은 종료 일시 + 높은 EXSEQ</span></span>
<span>    mode <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"last"</span>,                   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 마지막 기록 선택</span></span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Treatment Start and End Date (TRTSDT, TRTEDT 생성.)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dtm_to_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>source_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDTM</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTEDTM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Convert Datetime variables to date</span></span>
<span>      </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Treatment Start Time (TRTSM)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dtm_to_tm</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>source_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDTM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Treatment Duration ( TRTDURD = TRTEDT - TRTSDT + 1 )</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_var_trtdurd</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Safety Population Flag (SAFFL: "Y" or `NA`)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_var_merged_exist_flag</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span>,</span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    new_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">SAFFL</span>,</span>
<span>    condition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXTRT</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PLACEBO"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># adsl_raw = adsl_ct + 치료(exposure)관련 변수 및 SAFFL.</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_raw</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTSDTM"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTSTM"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTSTMF"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTSDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTEDTM"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTETMF"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTEDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTDURD"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SAFFL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-17b7c16e027c56386505" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-17b7c16e027c56386505">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["2014-01-02T00:00:00Z","2012-08-05T00:00:00Z","2013-07-19T00:00:00Z","2014-03-18T00:00:00Z","2014-07-01T00:00:00Z","2013-02-12T00:00:00Z"],["00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00"],["H","H","H","H","H","H"],["2014-01-02","2012-08-05","2013-07-19","2014-03-18","2014-07-01","2013-02-12"],["2014-07-02T23:59:59Z","2012-09-01T23:59:59Z","2014-01-14T23:59:59Z","2014-03-31T23:59:59Z","2014-12-30T23:59:59Z","2013-03-09T23:59:59Z"],["H","H","H","H","H","H"],["2014-07-02","2012-09-01","2014-01-14","2014-03-31","2014-12-30","2013-03-09"],[182,28,180,14,183,26],["Y","Y","Y","Y","Y","Y"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>TRTSDTM<\/th>\n      <th>TRTSTM<\/th>\n      <th>TRTSTMF<\/th>\n      <th>TRTSDT<\/th>\n      <th>TRTEDTM<\/th>\n      <th>TRTETMF<\/th>\n      <th>TRTEDT<\/th>\n      <th>TRTDURD<\/th>\n      <th>SAFFL<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":9},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"TRTSDTM","targets":2},{"name":"TRTSTM","targets":3},{"name":"TRTSTMF","targets":4},{"name":"TRTSDT","targets":5},{"name":"TRTEDTM","targets":6},{"name":"TRTETMF","targets":7},{"name":"TRTEDT","targets":8},{"name":"TRTDURD","targets":9},{"name":"SAFFL","targets":10}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br><strong>Derive Treatment Variables</strong></p>
<p>ADSL data에 치료 그룹 변수(<code>TRT01P</code>, <code>TRT01A</code>)와 숫자형 코드 변수(<code>TRT01PN</code>, <code>TRT01AN</code>)를 추가하자.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_raw</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    TRT01P <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ARM</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Screen Failure"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Assigned"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Treated"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"No Treatment"</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ARM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    TRT01A <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ACTARM</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Screen Failure"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Assigned"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Not Treated"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"No Treatment"</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ACTARM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">create_var_from_codelist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, input_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRT01P</span>, out_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRT01PN</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">create_var_from_codelist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, input_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRT01A</span>, out_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRT01AN</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><code>TRT01P</code>: 계획된 치료 그룹 (Planned Treatment) <br><code>TRT01A</code>: 실제 치료 그룹 (Actual Treatment)</p>
<pre><code>code mapping (example)
- "Placebo"     → 1
- "High Dose"   → 2
- "Low Dose"    → 3
- "No Treatment"→ NA</code></pre>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb16" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRT01P"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRT01PN"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRT01A"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRT01AN"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-7b45c1d662dbd1cb31c3" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-7b45c1d662dbd1cb31c3">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo"],[1,1,2,3,2,1],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo"],[1,1,2,3,2,1]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>TRT01P<\/th>\n      <th>TRT01PN<\/th>\n      <th>TRT01A<\/th>\n      <th>TRT01AN<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[4,6]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"TRT01P","targets":3},{"name":"TRT01PN","targets":4},{"name":"TRT01A","targets":5},{"name":"TRT01AN","targets":6}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br><strong>Derive Disposition Variables</strong></p>
<p>ADSL data에 연구 종료 변수(<code>EOSDT</code>, <code>EOSSTT</code>) 및 날짜 변수(<code>RANDDT</code>, <code>SCRFDT</code> 등)들을 추가하자.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb17" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1. 연구 종료 일자(EOSDT)</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># DS 데이터에서 문자형 날짜(DTC) → 숫자형 날짜(DT) 변환</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds_ext</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span>,</span>
<span>    dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSSTDTC</span>,           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 문자형 일자 (예: "2014-07-02")</span></span>
<span>    new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DSST"</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 생성 변수: DSSTDT, DSSTDTF</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#. ADSL에 연구 종료 일자 병합 (EOSDT, ex: 2014-07-02)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds_ext</span>,</span>
<span>      by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>EOSDT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSSTDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSCAT</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DISPOSITION EVENT"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSDECOD</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SCREEN FAILURE"</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 2. 연구 종료 상태(EOSSTT)</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사용자 정의 매핑 함수</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">format_eosstt</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">case_when</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"COMPLETED"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"COMPLETED"</span>,</span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SCREEN FAILURE"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA_character_</span>,</span>
<span>      <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DISCONTINUED"</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ADSL에 상태 변수 병합</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span>,</span>
<span>      by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSCAT</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DISPOSITION EVENT"</span>,</span>
<span>      new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>EOSSTT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">format_eosstt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSDECOD</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      missing_values <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>EOSSTT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ONGOING"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 결측 시 "ONGOING" 할당</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EOSDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EOSSTT"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-d8d1f074216943eca327" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-d8d1f074216943eca327">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["2014-07-02","2012-09-02","2014-01-14","2014-04-14","2014-12-30","2013-03-29"],["COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>EOSDT<\/th>\n      <th>EOSSTT<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"EOSDT","targets":3},{"name":"EOSSTT","targets":4}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<pre><code>    매핑 규칙:
    DSDECOD             EOSSTT
    "COMPLETED"         "COMPLETED"
    "SCREEN FAILURE"    NA
    기타                "DISCONTINUED"
    
    특이 사항: 처분 데이터가 없는 대상자는 "ONGOING"으로 표시</code></pre>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb19" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 3. 사망일자(DTHDT)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTH"</span>,      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 생성 변수: DTHDT, DTHDTF</span></span>
<span>      dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDTC</span>,                 <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사망 일자 (DTC)</span></span>
<span>      highest_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"M"</span>,     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 월까지만 있는 경우 "01"일로 대체</span></span>
<span>      date_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"first"</span>     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 일 결측 시 월의 첫째 날 대체</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 대체 규칙 (예시):</span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># "2014-07" → 2014-07-01</span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># "2014"    → 2014-01-01</span></span>
<span>  </span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4. 추가 날짜 변수  </span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 무작위 배정 일자(RANDDT)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds_ext</span>,</span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>RANDDT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSSTDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSDECOD</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RANDOMIZED"</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 스크린 실패 일자(SCRFDT)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds_ext</span>,</span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>SCRFDT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSSTDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSDECOD</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SCREEN FAILURE"</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 최종 방문 일자(FRVDT)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds_ext</span>,</span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>FRVDT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSSTDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSDECOD</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"FINAL RETRIEVAL VISIT"</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RANDDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SCRFDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"FRVDT"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-8b0281bfa9dbe7f1645f" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-8b0281bfa9dbe7f1645f">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["2014-01-02","2012-08-05","2013-07-19","2014-03-18","2014-07-01","2013-02-12"],[null,null,null,null,null,null],[null,"2013-02-18",null,"2014-09-15",null,"2013-07-28"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>RANDDT<\/th>\n      <th>SCRFDT<\/th>\n      <th>FRVDT<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"RANDDT","targets":3},{"name":"SCRFDT","targets":4},{"name":"FRVDT","targets":5}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb20" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 5. 사망 관련 기간 변수</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사망 상대일(DTHADY): 치료 시작일(TRTSDT) 기준</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_duration</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    new_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHADY</span>,</span>
<span>    start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDT</span>,</span>
<span>    end_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDT</span>,</span>
<span>    add_one <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 시작일 포함 (+1일)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 최종 투여-사망 간격(LDDTHELD): 치료 종료일(TRTEDT) 기준</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_duration</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    new_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">LDDTHELD</span>,</span>
<span>    start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTEDT</span>,</span>
<span>    end_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDT</span>,</span>
<span>    add_one <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 시작일 미포함</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># DTHADY    = DTHDT - TRTSDT + 1</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># LDDTHELD  = DTHDT - TRTEDT</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사용자 정의 함수</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">assign_randfl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">if_else</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">x</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Y"</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA_character_</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 플래그 할당</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>RANDFL <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">assign_randfl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RANDDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTSDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTEDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHADY"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"LDDTHELD"</span>,  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RANDDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RANDFL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-6351ecbf2d1e7aaccb4e" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-6351ecbf2d1e7aaccb4e">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["2014-01-02","2012-08-05","2013-07-19","2014-03-18","2014-07-01","2013-02-12"],["2014-07-02","2012-09-01","2014-01-14","2014-03-31","2014-12-30","2013-03-09"],[null,null,null,null,null,null],[null,null,null,null,null,null],["2014-01-02","2012-08-05","2013-07-19","2014-03-18","2014-07-01","2013-02-12"],["Y","Y","Y","Y","Y","Y"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>TRTSDT<\/th>\n      <th>TRTEDT<\/th>\n      <th>DTHADY<\/th>\n      <th>LDDTHELD<\/th>\n      <th>RANDDT<\/th>\n      <th>RANDFL<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[5,6]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"TRTSDT","targets":3},{"name":"TRTEDT","targets":4},{"name":"DTHADY","targets":5},{"name":"LDDTHELD","targets":6},{"name":"RANDDT","targets":7},{"name":"RANDFL","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br><strong>Derive Cause of Death</strong></p>
<p><code>admiral::derive_vars_extreme_event()</code>를 이용, <br> dm/suppdm이 아닌 다른 위치(AE/DS 등)의 사망 원인 정보를 통합, ADSL에 추가할 수 있다.</p>
<ul>
<li>
<code>DTHDT</code> : death date</li>
<li>
<code>DTHCAUS</code> : death cause</li>
<li>
<code>DTHDOM</code> : death domain (ex: “AE”/“DS”)</li>
</ul>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb21" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_extreme_event</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 그룹화 변수</span></span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사망 원인 이벤트 정의</span></span>
<span>    events <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1. AE 도메인에서 사망 원인 추출</span></span>
<span>      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">event</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>        dataset_name <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ae"</span>,</span>
<span>        condition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AEOUT</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"FATAL"</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치명적 부작용 필터</span></span>
<span>        set_values_to <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>          DTHCAUS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AEDECOD</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 부작용 용어 사용</span></span>
<span>          DTHDOM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AE"</span>       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 출처 도메인 표시</span></span>
<span>        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 2. DS 도메인에서 사망 원인 추출</span></span>
<span>      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">event</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>        dataset_name <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ds"</span>,</span>
<span>        condition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSDECOD</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DEATH"</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/grep.html">grepl</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DEATH DUE TO"</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSTERM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>        set_values_to <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>          DTHCAUS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DSTERM</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 처분 용어 사용</span></span>
<span>          DTHDOM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DS"</span>       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 출처 도메인 표시</span></span>
<span>        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 소스 데이터셋 지정</span></span>
<span>    source_datasets <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>ae <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span>, ds <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ds</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 기술적 옵션</span></span>
<span>    tmp_event_nr_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">event_nr</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 임시 이벤트 번호 변수</span></span>
<span>    order <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">event_nr</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 정렬 기준 (첫 번째 이벤트 선택)</span></span>
<span>    mode <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"first"</span>,               <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 최초 발생 이벤트 우선</span></span>
<span></span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 출력 변수</span></span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHCAUS</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDOM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/filter.html">filter</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHCAUS"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHDOM"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-fdc11ebe39922c099c7e" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-fdc11ebe39922c099c7e">{"x":{"filter":"none","vertical":false,"data":[["1","2","3"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1211","01-704-1445","01-710-1083"],["2013-01-14","2014-11-01","2013-08-02"],["SUDDEN DEATH","COMPLETED SUICIDE","MYOCARDIAL INFARCTION"],["AE","AE","AE"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>DTHDT<\/th>\n      <th>DTHCAUS<\/th>\n      <th>DTHDOM<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"DTHDT","targets":3},{"name":"DTHCAUS","targets":4},{"name":"DTHDOM","targets":5}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br><strong>Derive Other Grouping Variables</strong></p>
<ul>
<li>지역, 인종, 사망 범주 등 다양한 grouping variable들을 추가할 수 있다.</li>
<li>
<code>derive_vars_cat()</code>을 이용, factor와 numeric variable 추가을 추가할 수 있다.</li>
</ul>
<pre><code>주요 변수
변수명    설명                   생성 규칙 예시
REGION1   지역 그룹 (문자형)   북미 vs. 기타 국가
REGION1N  지역 코드 (숫자형)   1=North America, 2=Rest of the World
RACEGR1   인종 그룹 (문자형)   White vs. Non-white
RACEGR1N  인종 코드 (숫자형)   1=White, 2=Non-white
DTHCGR1   사망 원인 그룹       Adverse Event vs. Progressive Disease
DTHCGR1N  사망 원인 코드       1=ADVERSE EVENT, 2=PROGRESSIVE DISEASE</code></pre>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb23" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># REGION1</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">region1_lookup</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">condition</span>,                     <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">REGION1</span>,          <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">REGION1N</span>,</span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">COUNTRY</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"CAN"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USA"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,   <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"North America"</span>,     <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">COUNTRY</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rest of the World"</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,</span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">COUNTRY</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,                 <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Missing"</span>,           <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># RACEGR1</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">racegr1_lookup</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">condition</span>,        <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACEGR1</span>,    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACEGR1N</span>,</span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"WHITE"</span>,   <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"White"</span>,     <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"WHITE"</span>,   <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Non-white"</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,</span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">RACE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,       <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Missing"</span>,   <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#DTHCG1</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dthcgr1_lookup</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">condition</span>,                                                   <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHCGR1</span>,              <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHCGR1N</span>,</span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDOM</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AE"</span>,                                               <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADVERSE EVENT"</span>,       <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,</span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHCAUS</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"(PROGRESSIVE DISEASE|DISEASE RELAPSE)"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PROGRESSIVE DISEASE"</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,</span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHCAUS</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,                                              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"OTHER"</span>,               <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>,</span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDOM</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,                                                <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA_character_</span>,         <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 변수 생성</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_cat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>definition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">region1_lookup</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_cat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>definition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">racegr1_lookup</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_cat</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>definition <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dthcgr1_lookup</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">head</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"COUNTRY"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"REGION1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"REGION1N"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RACE"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RACEGR1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RACEGR1N"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-7711d7ddd13c45d68eac" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-7711d7ddd13c45d68eac">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047"],["USA","USA","USA","USA","USA","USA"],["North America","North America","North America","North America","North America","North America"],[1,1,1,1,1,1],["WHITE","WHITE","WHITE","WHITE","WHITE","WHITE"],["White","White","White","White","White","White"],[1,1,1,1,1,1]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>COUNTRY<\/th>\n      <th>REGION1<\/th>\n      <th>REGION1N<\/th>\n      <th>RACE<\/th>\n      <th>RACEGR1<\/th>\n      <th>RACEGR1N<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[5,8]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"COUNTRY","targets":3},{"name":"REGION1","targets":4},{"name":"REGION1N","targets":5},{"name":"RACE","targets":6},{"name":"RACEGR1","targets":7},{"name":"RACEGR1N","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb24" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/filter.html">filter</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"STUDYID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHDOM"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHCAUS"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHCGR1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"DTHCGR1N"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-9b3ade22ab0907edea7b" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-9b3ade22ab0907edea7b">{"x":{"filter":"none","vertical":false,"data":[["1","2","3"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1211","01-704-1445","01-710-1083"],["AE","AE","AE"],["SUDDEN DEATH","COMPLETED SUICIDE","MYOCARDIAL INFARCTION"],["ADVERSE EVENT","ADVERSE EVENT","ADVERSE EVENT"],[1,1,1]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>DTHDOM<\/th>\n      <th>DTHCAUS<\/th>\n      <th>DTHCGR1<\/th>\n      <th>DTHCGR1N<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":6},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"DTHDOM","targets":3},{"name":"DTHCAUS","targets":4},{"name":"DTHCGR1","targets":5},{"name":"DTHCGR1N","targets":6}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br><strong>Data check, create eSub XPT</strong></p>
<p>ADSL data에 metadata를 적용, 검토해보자. 검토와 동시에 최종 결과를 확인할 수 있다. <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb25" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dir</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/tempfile.html">tempdir</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1. 변수 존재 여부 검증</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">check_variables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 메타데이터에 정의된 변수 모두 존재하는지 확인</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 2. 코드리스트(Controlled Terminology) 검증</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">check_ct_data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, na_acceptable <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 허용된 값만 포함하는지 확인 (NA 허용)</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 3. 컬럼 순서 정렬</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">order_cols</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 메타데이터 스펙에 정의된 순서로 재배열</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4. 키 기준 행 정렬</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sort_by_key</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 메타데이터의 sort_key 변수(예: USUBJID) 기준 정렬</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 5. eSub XPT로 저장</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, domain <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADSL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 변수 타입 강제 변환 (예: 문자 → 숫자)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_length</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>                 <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># SAS 길이 지정 (예: 문자열 200자)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>                  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 변수 라벨 할당 (예: "Age at Baseline")</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_df_label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>               <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 데이터셋 라벨 할당 (예: "Adverse Events Analysis Dataset")</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_write</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/file.path.html">file.path</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dir</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"adsl.xpt"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, metadata <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, domain <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADSL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-2cfe3e34439f1460dc10" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-2cfe3e34439f1460dc10">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144","145","146","147","148","149","150","151","152","153","154","155","156","157","158","159","160","161","162","163","164","165","166","167","168","169","170","171","172","173","174","175","176","177","178","179","180","181","182","183","184","185","186","187","188","189","190","191","192","193","194","195","196","197","198","199","200","201","202","203","204","205","206","207","208","209","210","211","212","213","214","215","216","217","218","219","220","221","222","223","224","225","226","227","228","229","230","231","232","233","234","235","236","237","238","239","240","241","242","243","244","245","246","247","248","249","250","251","252","253","254","255","256","257","258","259","260","261","262","263","264","265","266","267","268","269","270","271","272","273","274","275","276","277","278","279","280","281","282","283","284","285","286","287","288","289","290","291","292","293","294","295","296","297","298","299","300","301","302","303","304","305","306"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1023","01-701-1028","01-701-1033","01-701-1034","01-701-1047","01-701-1057","01-701-1097","01-701-1111","01-701-1115","01-701-1118","01-701-1130","01-701-1133","01-701-1145","01-701-1146","01-701-1148","01-701-1153","01-701-1162","01-701-1176","01-701-1180","01-701-1181","01-701-1188","01-701-1192","01-701-1203","01-701-1211","01-701-1234","01-701-1239","01-701-1240","01-701-1275","01-701-1287","01-701-1294","01-701-1302","01-701-1307","01-701-1317","01-701-1324","01-701-1341","01-701-1345","01-701-1356","01-701-1360","01-701-1363","01-701-1369","01-701-1383","01-701-1386","01-701-1387","01-701-1392","01-701-1411","01-701-1415","01-701-1429","01-701-1440","01-701-1442","01-701-1444","01-702-1082","01-703-1042","01-703-1076","01-703-1086","01-703-1096","01-703-1100","01-703-1119","01-703-1175","01-703-1182","01-703-1197","01-703-1210","01-703-1258","01-703-1279","01-703-1295","01-703-1299","01-703-1335","01-703-1379","01-703-1396","01-703-1403","01-703-1439","01-704-1008","01-704-1009","01-704-1010","01-704-1017","01-704-1025","01-704-1065","01-704-1074","01-704-1093","01-704-1114","01-704-1120","01-704-1127","01-704-1135","01-704-1164","01-704-1218","01-704-1233","01-704-1241","01-704-1260","01-704-1266","01-704-1323","01-704-1325","01-704-1332","01-704-1351","01-704-1388","01-704-1435","01-704-1445","01-705-1011","01-705-1018","01-705-1031","01-705-1058","01-705-1059","01-705-1112","01-705-1186","01-705-1199","01-705-1243","01-705-1280","01-705-1281","01-705-1282","01-705-1292","01-705-1303","01-705-1310","01-705-1349","01-705-1377","01-705-1382","01-705-1393","01-705-1421","01-705-1431","01-706-1041","01-706-1049","01-706-1384","01-707-1037","01-707-1206","01-707-1276","01-707-1430","01-707-1434","01-708-1013","01-708-1019","01-708-1032","01-708-1054","01-708-1067","01-708-1084","01-708-1087","01-708-1104","01-708-1158","01-708-1171","01-708-1178","01-708-1184","01-708-1213","01-708-1216","01-708-1236","01-708-1242","01-708-1253","01-708-1272","01-708-1286","01-708-1296","01-708-1297","01-708-1316","01-708-1336","01-708-1342","01-708-1347","01-708-1348","01-708-1352","01-708-1353","01-708-1372","01-708-1378","01-708-1406","01-708-1428","01-709-1001","01-709-1007","01-709-1020","01-709-1029","01-709-1081","01-709-1088","01-709-1099","01-709-1102","01-709-1168","01-709-1217","01-709-1224","01-709-1237","01-709-1238","01-709-1259","01-709-1285","01-709-1301","01-709-1306","01-709-1309","01-709-1312","01-709-1326","01-709-1329","01-709-1339","01-709-1424","01-710-1002","01-710-1006","01-710-1021","01-710-1027","01-710-1045","01-710-1053","01-710-1060","01-710-1070","01-710-1077","01-710-1078","01-710-1083","01-710-1129","01-710-1137","01-710-1142","01-710-1149","01-710-1154","01-710-1166","01-710-1183","01-710-1187","01-710-1235","01-710-1249","01-710-1257","01-710-1264","01-710-1270","01-710-1271","01-710-1278","01-710-1300","01-710-1314","01-710-1315","01-710-1337","01-710-1354","01-710-1358","01-710-1368","01-710-1376","01-710-1380","01-710-1385","01-710-1408","01-710-1443","01-711-1012","01-711-1022","01-711-1036","01-711-1143","01-711-1163","01-711-1173","01-711-1226","01-711-1251","01-711-1283","01-711-1284","01-711-1290","01-711-1433","01-713-1043","01-713-1073","01-713-1106","01-713-1141","01-713-1179","01-713-1209","01-713-1256","01-713-1269","01-713-1448","01-714-1035","01-714-1068","01-714-1195","01-714-1288","01-714-1375","01-714-1425","01-715-1085","01-715-1107","01-715-1134","01-715-1155","01-715-1207","01-715-1225","01-715-1319","01-715-1321","01-715-1333","01-715-1397","01-715-1405","01-715-1407","01-716-1003","01-716-1024","01-716-1026","01-716-1030","01-716-1044","01-716-1061","01-716-1063","01-716-1071","01-716-1094","01-716-1103","01-716-1108","01-716-1151","01-716-1157","01-716-1160","01-716-1167","01-716-1177","01-716-1189","01-716-1229","01-716-1244","01-716-1298","01-716-1305","01-716-1308","01-716-1311","01-716-1331","01-716-1364","01-716-1373","01-716-1418","01-716-1441","01-716-1447","01-717-1004","01-717-1109","01-717-1174","01-717-1201","01-717-1344","01-717-1357","01-717-1446","01-718-1066","01-718-1079","01-718-1101","01-718-1139","01-718-1150","01-718-1170","01-718-1172","01-718-1250","01-718-1254","01-718-1328","01-718-1355","01-718-1371","01-718-1427"],["1015","1023","1028","1033","1034","1047","1057","1097","1111","1115","1118","1130","1133","1145","1146","1148","1153","1162","1176","1180","1181","1188","1192","1203","1211","1234","1239","1240","1275","1287","1294","1302","1307","1317","1324","1341","1345","1356","1360","1363","1369","1383","1386","1387","1392","1411","1415","1429","1440","1442","1444","1082","1042","1076","1086","1096","1100","1119","1175","1182","1197","1210","1258","1279","1295","1299","1335","1379","1396","1403","1439","1008","1009","1010","1017","1025","1065","1074","1093","1114","1120","1127","1135","1164","1218","1233","1241","1260","1266","1323","1325","1332","1351","1388","1435","1445","1011","1018","1031","1058","1059","1112","1186","1199","1243","1280","1281","1282","1292","1303","1310","1349","1377","1382","1393","1421","1431","1041","1049","1384","1037","1206","1276","1430","1434","1013","1019","1032","1054","1067","1084","1087","1104","1158","1171","1178","1184","1213","1216","1236","1242","1253","1272","1286","1296","1297","1316","1336","1342","1347","1348","1352","1353","1372","1378","1406","1428","1001","1007","1020","1029","1081","1088","1099","1102","1168","1217","1224","1237","1238","1259","1285","1301","1306","1309","1312","1326","1329","1339","1424","1002","1006","1021","1027","1045","1053","1060","1070","1077","1078","1083","1129","1137","1142","1149","1154","1166","1183","1187","1235","1249","1257","1264","1270","1271","1278","1300","1314","1315","1337","1354","1358","1368","1376","1380","1385","1408","1443","1012","1022","1036","1143","1163","1173","1226","1251","1283","1284","1290","1433","1043","1073","1106","1141","1179","1209","1256","1269","1448","1035","1068","1195","1288","1375","1425","1085","1107","1134","1155","1207","1225","1319","1321","1333","1397","1405","1407","1003","1024","1026","1030","1044","1061","1063","1071","1094","1103","1108","1151","1157","1160","1167","1177","1189","1229","1244","1298","1305","1308","1311","1331","1364","1373","1418","1441","1447","1004","1109","1174","1201","1344","1357","1446","1066","1079","1101","1139","1150","1170","1172","1250","1254","1328","1355","1371","1427"],["701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","702","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","706","706","706","707","707","707","707","707","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","711","711","711","711","711","711","711","711","711","711","711","711","713","713","713","713","713","713","713","713","713","714","714","714","714","714","714","715","715","715","715","715","715","715","715","715","715","715","715","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","717","717","717","717","717","717","717","718","718","718","718","718","718","718","718","718","718","718","718","718"],["USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA"],["North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America","North America"],[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],[63,64,71,74,77,85,59,68,81,84,52,84,81,57,75,57,79,82,62,56,79,71,80,81,76,69,56,57,61,56,67,61,80,68,79,51,63,54,67,81,74,72,71,87,78,76,85,84,70,57,63,84,64,69,71,81,84,81,75,84,76,72,78,72,88,81,67,81,84,67,76,76,83,80,77,81,75,80,79,77,71,84,74,67,81,87,86,71,82,68,81,80,70,81,74,75,75,69,56,89,66,78,84,87,70,56,73,70,60,72,74,86,63,82,84,87,68,64,60,74,72,65,72,81,76,80,68,62,73,88,73,74,78,81,77,77,70,76,78,86,80,61,82,80,57,61,74,73,59,61,79,87,87,84,67,71,84,76,54,72,82,86,69,79,71,72,77,84,88,69,82,87,62,60,65,68,75,70,81,77,88,77,79,83,83,84,82,85,76,81,89,69,79,76,79,84,81,80,78,56,79,85,78,83,86,81,78,78,83,73,73,82,88,89,80,77,80,88,67,86,70,76,74,62,74,82,75,77,61,84,78,74,74,79,64,77,71,73,71,88,79,75,77,78,81,77,65,50,59,78,77,65,75,75,76,69,73,79,87,73,83,74,81,80,78,82,79,86,83,85,83,68,72,81,73,72,76,73,76,78,59,84,74,80,85,72,80,84,73,85,64,77,75,79,67,82,77,73,80,74,82,78,86,79,69,74],["YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS"],["18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","18-64","&gt;64","18-64","&gt;64","&gt;64","18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","18-64","18-64","18-64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","18-64","18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","18-64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","18-64","18-64","&gt;64","&gt;64","18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","18-64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64","&gt;64"],[2,2,3,3,3,3,2,3,3,3,2,3,3,2,3,2,3,3,2,2,3,3,3,3,3,3,2,2,2,2,3,2,3,3,3,2,2,2,3,3,3,3,3,3,3,3,3,3,3,2,2,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,2,3,3,2,3,3,3,2,3,3,3,3,2,2,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,3,2,2,3,3,2,2,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,3,3,3,2,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3],["F","M","M","M","F","F","F","M","F","M","M","M","F","F","F","M","F","F","F","M","F","M","F","F","F","M","M","F","M","F","M","M","F","M","M","M","F","F","M","F","M","F","M","F","M","M","M","F","M","F","M","F","M","M","M","F","F","F","M","M","F","F","F","F","F","F","F","F","M","M","F","F","M","M","M","F","M","F","M","M","F","F","F","F","F","F","M","F","M","F","M","M","M","M","M","M","F","F","F","M","F","M","F","M","F","F","F","F","F","M","F","F","F","M","F","M","F","F","F","F","F","M","F","F","F","M","M","M","F","M","F","F","F","F","F","F","F","F","M","F","M","M","M","F","M","M","F","M","F","F","F","F","F","M","M","F","F","F","F","F","M","F","M","F","F","F","M","M","F","M","M","M","F","F","M","F","F","M","M","M","M","M","M","M","F","F","M","F","F","F","F","F","F","F","F","M","F","F","F","F","M","F","M","F","F","M","F","F","F","M","M","M","F","F","F","M","M","F","F","F","M","F","F","M","F","M","F","F","F","F","F","F","M","M","F","F","M","M","F","F","F","M","M","F","M","F","M","F","F","F","F","M","F","F","F","M","F","M","F","F","F","M","M","M","F","M","M","F","F","M","F","M","M","M","F","F","F","F","F","M","F","F","M","F","M","F","F","M","M","F","F","M","F","F","F","M","M","F","F","M","F","M","M","M","F","F"],["WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","AMERICAN INDIAN OR ALASKA NATIVE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","AMERICAN INDIAN OR ALASKA NATIVE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","ASIAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","ASIAN","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN"],["Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y",null,"Y","Y","Y","Y",null,"Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y",null,null,"Y","Y",null,"Y","Y","Y",null,"Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y",null,null,"Y","Y",null,"Y",null,"Y","Y",null,null,null,null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y","Y",null,"Y","Y",null,null,"Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y"],[4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,1,4,4,4,4,3,4,4,4,3,1,4,4,4,4,4,4,4,4,4,4,3,4,4,4,4,4,3,4,4,4,3,4,4,4,4,4,4,4,4,4,4,3,4,4,4,4,4,3,3,2,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,3,3,3,4,4,4,3,4,4,4,4,3,4,4,4,4,4,4,4,4,4,4,3,4,4,4,2,4,4,3,4,4,4,4,4,4,4,3,3,4,4,4,4,4,4,4,3,4,3,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,3,4,4,4,4,4,4,4,4,3,3,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,3,4,3,4,4,4,3,4,4,4,4,3,4,4,4,4,4,4,4,4,4,4,3,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,3,4,4,4,4,4,4,4,4,4,3],["White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","White","Non-white","White","White","White","Non-white","Non-white","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","White","White","Non-white","White","White","White","Non-white","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","White","White","Non-white","Non-white","Non-white","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","Non-white","Non-white","Non-white","White","White","White","Non-white","White","White","White","White","Non-white","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","Non-white","White","White","Non-white","White","White","White","White","White","White","White","Non-white","Non-white","White","White","White","White","White","White","White","Non-white","White","Non-white","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","White","White","White","White","White","Non-white","Non-white","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","Non-white","White","Non-white","White","White","White","Non-white","White","White","White","White","Non-white","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","White","Non-white","White","White","White","White","White","White","White","White","White","Non-white"],[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,2,1,1,1,2,2,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,2,1,1,1,2,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,2,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,1,1,1,2,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,2,1,1,1,2,1,1,2,1,1,1,1,1,1,1,2,2,1,1,1,1,1,1,1,2,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,2,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,2,1,1,1,2,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,2],["HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO"],["Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y",null,"Y","Y","Y","Y",null,"Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y",null,null,"Y","Y",null,"Y","Y","Y",null,"Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y",null,null,"Y","Y",null,"Y",null,"Y","Y",null,null,null,null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y","Y",null,"Y","Y",null,null,"Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y"],["Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y",null,"Y","Y","Y","Y",null,"Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y",null,null,"Y","Y",null,"Y","Y","Y",null,"Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y",null,null,"Y","Y",null,"Y",null,"Y","Y",null,null,null,null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,"Y","Y",null,"Y","Y",null,null,"Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y",null,"Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y"],["2014-01-02","2012-08-05","2013-07-19","2014-03-18","2014-07-01","2013-02-12",null,"2014-01-01","2012-09-07","2012-11-30","2014-03-12","2014-02-15","2012-10-28",null,"2013-05-20","2013-08-23","2013-09-23",null,null,"2013-02-12","2013-12-05","2013-02-15","2012-07-22","2013-02-02","2012-11-15","2013-03-30","2014-01-11",null,"2014-02-07","2014-01-25","2013-03-24","2013-08-29",null,"2014-05-22","2012-10-02","2013-01-05","2013-10-08",null,"2013-07-31","2013-05-30",null,"2013-02-04",null,"2014-03-12","2012-10-28",null,"2013-09-23","2013-03-19","2013-08-08","2013-10-26","2013-01-05","2013-07-26","2013-03-02","2013-10-25","2012-09-02","2013-01-25","2013-03-13","2013-02-20","2013-12-20","2013-10-17","2013-06-16","2013-03-16","2012-07-20","2013-05-13","2013-11-21","2012-09-12","2014-03-17","2013-09-22",null,"2012-12-12","2014-03-12","2013-01-13","2013-08-27","2014-02-21","2013-10-06","2013-09-27","2013-10-24","2014-01-22","2013-03-15","2013-01-23","2013-12-02","2013-10-02","2013-10-31","2012-09-19","2012-11-19","2013-03-21","2013-08-25","2012-08-30","2013-10-13","2013-07-08","2014-04-23","2013-12-09","2013-10-12","2012-12-07","2012-11-17","2014-05-11",null,"2013-07-05","2013-11-27",null,"2013-08-05",null,"2014-01-08","2013-09-16",null,"2014-01-17","2013-11-28","2012-12-26","2013-10-14","2013-12-16","2013-11-02","2013-03-10","2014-01-04","2013-05-13","2012-09-07",null,"2013-06-23","2013-12-31","2013-05-14","2012-09-15","2013-12-20","2013-10-28",null,null,null,null,"2013-12-20","2013-02-09",null,null,"2013-05-09","2012-10-22",null,"2014-02-08","2012-12-06","2013-12-29",null,"2013-02-09","2012-10-24","2013-09-21",null,"2013-05-07","2013-02-06","2013-09-10","2013-06-14","2013-01-25","2013-08-23","2012-12-07","2012-12-29","2013-04-20","2013-08-05",null,"2013-07-04","2013-04-12","2013-09-03","2013-12-26","2013-11-09","2013-10-08","2012-07-31","2012-12-01","2012-12-25","2014-01-18","2014-04-12","2013-10-25","2013-01-15","2013-08-02","2013-03-04",null,null,"2013-05-15","2013-01-26","2013-03-24","2013-07-19","2014-02-03","2013-06-19","2014-04-20","2013-04-05","2013-08-15","2012-12-23","2013-03-03","2014-01-14","2013-02-10","2013-09-27","2014-02-28","2013-06-03","2012-12-26","2013-01-01","2012-09-08","2013-11-17","2013-09-17","2013-07-22",null,"2013-10-11","2012-10-02",null,"2014-03-29","2012-11-30","2013-11-16","2012-11-10","2012-09-26","2013-12-28",null,"2013-06-13","2014-02-12","2012-09-27","2012-12-24","2012-12-15","2013-04-24","2013-02-27",null,"2012-11-11","2012-09-17","2013-10-23",null,null,"2012-10-29","2013-01-05",null,"2013-04-03",null,"2012-07-29","2013-04-03",null,null,null,null,null,null,null,"2013-01-27","2013-09-15","2014-03-30","2012-10-31","2013-05-31","2013-08-09","2013-05-27","2012-09-19","2014-01-27","2014-01-19","2014-04-17","2013-08-08","2013-04-26","2013-12-04","2013-02-26","2013-08-15","2013-02-16","2013-02-26",null,"2013-12-13","2012-11-18",null,"2013-02-17","2014-02-11",null,"2013-04-03","2013-07-06",null,null,"2012-07-09","2014-04-02","2013-12-28","2013-04-27",null,"2013-05-09","2013-06-08","2012-12-19","2014-03-20","2013-02-12","2013-02-01","2013-10-02","2013-04-05","2012-10-08","2014-09-02","2012-10-09","2013-02-20",null,"2013-04-08",null,"2013-08-28","2014-05-14",null,"2013-07-04","2012-12-14","2013-05-05","2014-01-22","2013-12-16","2014-01-14","2014-01-27","2013-01-22","2013-12-19","2014-01-11","2013-05-01","2013-09-01","2013-07-07","2012-09-19","2013-02-17","2013-05-19","2013-01-19","2013-09-16","2013-09-21","2013-09-21","2013-07-10","2013-02-01","2013-02-28","2013-04-26","2012-12-17"],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Placebo","Screen Failure","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Screen Failure","Xanomeline High Dose","Placebo","Screen Failure","Xanomeline High Dose","Screen Failure","Placebo","Placebo","Screen Failure","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Placebo","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Screen Failure","Screen Failure","Xanomeline Low Dose","Placebo","Screen Failure","Placebo","Placebo","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Screen Failure","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Placebo","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","Screen Failure","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Screen Failure","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Screen Failure","Placebo","Placebo","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Screen Failure","Placebo","Placebo","Xanomeline High Dose","Placebo","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose"],["Pbo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Pbo","Scrnfail","Xan_Lo","Xan_Lo","Xan_Lo","Pbo","Pbo","Xan_Hi","Scrnfail","Xan_Hi","Xan_Hi","Pbo","Scrnfail","Scrnfail","Xan_Hi","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Hi","Scrnfail","Xan_Hi","Xan_Hi","Xan_Lo","Xan_Hi","Scrnfail","Xan_Lo","Xan_Lo","Xan_Lo","Pbo","Scrnfail","Xan_Hi","Pbo","Scrnfail","Xan_Hi","Scrnfail","Pbo","Pbo","Scrnfail","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Hi","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Pbo","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Scrnfail","Xan_Hi","Xan_Hi","Xan_Hi","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Hi","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Pbo","Pbo","Pbo","Scrnfail","Pbo","Xan_Lo","Scrnfail","Pbo","Scrnfail","Pbo","Xan_Lo","Scrnfail","Xan_Hi","Xan_Hi","Pbo","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Xan_Hi","Xan_Hi","Xan_Lo","Scrnfail","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Xan_Lo","Xan_Lo","Scrnfail","Scrnfail","Xan_Lo","Pbo","Scrnfail","Pbo","Pbo","Xan_Hi","Scrnfail","Xan_Hi","Xan_Hi","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Pbo","Pbo","Xan_Lo","Pbo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Scrnfail","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Lo","Scrnfail","Scrnfail","Xan_Hi","Pbo","Xan_Lo","Pbo","Pbo","Xan_Hi","Pbo","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Pbo","Pbo","Pbo","Scrnfail","Xan_Hi","Xan_Hi","Scrnfail","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Pbo","Pbo","Scrnfail","Xan_Hi","Xan_Lo","Pbo","Scrnfail","Scrnfail","Xan_Lo","Xan_Hi","Scrnfail","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Xan_Hi","Xan_Lo","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Xan_Hi","Pbo","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Scrnfail","Pbo","Pbo","Scrnfail","Xan_Hi","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Scrnfail","Scrnfail","Pbo","Pbo","Xan_Hi","Pbo","Scrnfail","Xan_Lo","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Hi","Xan_Hi","Scrnfail","Xan_Lo","Scrnfail","Pbo","Xan_Lo","Scrnfail","Xan_Hi","Xan_Hi","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Hi"],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Placebo","Screen Failure","Screen Failure","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Screen Failure","Xanomeline Low Dose","Placebo","Screen Failure","Xanomeline High Dose","Screen Failure","Placebo","Placebo","Screen Failure","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Placebo","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Screen Failure","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Screen Failure","Screen Failure","Xanomeline Low Dose","Placebo","Screen Failure","Placebo","Placebo","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Screen Failure","Screen Failure","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Placebo","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","Screen Failure","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Screen Failure","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Screen Failure","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Screen Failure","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Screen Failure","Placebo","Placebo","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Screen Failure","Placebo","Placebo","Xanomeline Low Dose","Placebo","Screen Failure","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Screen Failure","Xanomeline Low Dose","Screen Failure","Placebo","Xanomeline Low Dose","Screen Failure","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose"],["Pbo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Pbo","Scrnfail","Xan_Lo","Xan_Lo","Xan_Lo","Pbo","Pbo","Xan_Hi","Scrnfail","Xan_Hi","Xan_Hi","Pbo","Scrnfail","Scrnfail","Xan_Hi","Xan_Lo","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Hi","Scrnfail","Xan_Hi","Xan_Hi","Xan_Lo","Xan_Hi","Scrnfail","Xan_Lo","Xan_Lo","Xan_Lo","Pbo","Scrnfail","Xan_Lo","Pbo","Scrnfail","Xan_Hi","Scrnfail","Pbo","Pbo","Scrnfail","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Hi","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Pbo","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Scrnfail","Xan_Lo","Xan_Hi","Xan_Hi","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Hi","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Pbo","Pbo","Pbo","Scrnfail","Pbo","Xan_Lo","Scrnfail","Pbo","Scrnfail","Pbo","Xan_Lo","Scrnfail","Xan_Hi","Xan_Hi","Pbo","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Scrnfail","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Xan_Lo","Xan_Lo","Scrnfail","Scrnfail","Xan_Lo","Pbo","Scrnfail","Pbo","Pbo","Xan_Hi","Scrnfail","Xan_Lo","Xan_Hi","Xan_Lo","Scrnfail","Pbo","Xan_Lo","Pbo","Pbo","Xan_Lo","Pbo","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Scrnfail","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Lo","Scrnfail","Scrnfail","Xan_Hi","Pbo","Xan_Lo","Pbo","Pbo","Xan_Hi","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Pbo","Pbo","Pbo","Scrnfail","Xan_Hi","Xan_Hi","Scrnfail","Xan_Lo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Pbo","Xan_Hi","Xan_Lo","Pbo","Pbo","Scrnfail","Xan_Hi","Xan_Lo","Pbo","Scrnfail","Scrnfail","Xan_Lo","Xan_Hi","Scrnfail","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Scrnfail","Xan_Lo","Xan_Lo","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Xan_Hi","Pbo","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Xan_Lo","Xan_Lo","Xan_Lo","Scrnfail","Pbo","Pbo","Scrnfail","Xan_Hi","Xan_Hi","Scrnfail","Pbo","Xan_Lo","Scrnfail","Scrnfail","Pbo","Pbo","Xan_Lo","Pbo","Scrnfail","Xan_Lo","Xan_Hi","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Pbo","Xan_Lo","Pbo","Xan_Hi","Xan_Hi","Scrnfail","Xan_Lo","Scrnfail","Pbo","Xan_Lo","Scrnfail","Xan_Hi","Xan_Hi","Xan_Hi","Pbo","Xan_Hi","Xan_Lo","Xan_Hi","Xan_Hi","Pbo","Pbo","Xan_Hi","Xan_Lo","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Pbo","Xan_Lo","Pbo","Xan_Lo","Xan_Lo","Xan_Hi","Pbo","Xan_Hi","Xan_Hi"],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Placebo","No Treatment","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","No Treatment","Xanomeline High Dose","Placebo","No Treatment","Xanomeline High Dose","No Treatment","Placebo","Placebo","No Treatment","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","Placebo","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","No Treatment","No Treatment","No Treatment","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","No Treatment","No Treatment","Xanomeline Low Dose","Placebo","No Treatment","Placebo","Placebo","Xanomeline High Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","No Treatment","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Placebo","No Treatment","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","No Treatment","Xanomeline High Dose","Xanomeline Low Dose","Placebo","No Treatment","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","No Treatment","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","No Treatment","No Treatment","No Treatment","No Treatment","No Treatment","No Treatment","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","No Treatment","Placebo","Placebo","No Treatment","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","No Treatment","Placebo","Placebo","Xanomeline High Dose","Placebo","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose"],[1,1,2,3,2,1,888,3,3,3,1,1,2,888,2,2,1,888,888,2,2,3,3,1,3,1,2,888,2,2,3,2,888,3,3,3,1,888,2,1,888,2,888,1,1,888,1,3,1,3,2,3,1,2,3,1,1,3,1,3,3,1,2,3,2,1,2,3,888,2,2,2,3,1,2,3,2,2,2,3,3,1,3,1,3,1,2,1,2,3,3,2,1,1,1,1,888,1,3,888,1,888,1,3,888,2,2,1,3,2,2,1,2,2,3,888,3,1,2,3,3,1,888,888,888,888,3,3,888,888,3,1,888,1,1,2,888,2,2,2,888,1,3,1,1,3,1,2,1,2,3,888,3,2,1,2,3,1,3,3,2,3,1,2,3,2,3,888,888,2,1,3,1,1,2,1,3,2,1,2,3,2,2,1,3,3,1,2,1,1,1,888,2,2,888,3,3,1,2,3,2,888,1,3,1,2,3,1,1,888,2,3,1,888,888,3,2,888,2,888,1,3,888,888,888,888,888,888,888,2,3,3,2,2,1,2,1,1,3,1,3,3,2,1,2,3,3,888,1,1,888,2,2,888,1,3,888,888,1,1,2,1,888,3,2,3,3,1,3,3,1,3,1,2,2,888,3,888,1,3,888,2,2,2,1,2,3,2,2,1,1,2,3,3,3,2,1,1,3,1,3,3,2,1,2,2],["Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Placebo","No Treatment","No Treatment","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","No Treatment","Xanomeline Low Dose","Placebo","No Treatment","Xanomeline High Dose","No Treatment","Placebo","Placebo","No Treatment","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","Placebo","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","No Treatment","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","No Treatment","No Treatment","No Treatment","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","No Treatment","No Treatment","Xanomeline Low Dose","Placebo","No Treatment","Placebo","Placebo","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","No Treatment","No Treatment","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Placebo","No Treatment","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Placebo","Placebo","No Treatment","Xanomeline High Dose","Xanomeline Low Dose","Placebo","No Treatment","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","No Treatment","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","No Treatment","No Treatment","No Treatment","No Treatment","No Treatment","No Treatment","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","No Treatment","Placebo","Placebo","No Treatment","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","No Treatment","Placebo","Placebo","Xanomeline Low Dose","Placebo","No Treatment","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","No Treatment","Xanomeline Low Dose","No Treatment","Placebo","Xanomeline Low Dose","No Treatment","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose"],[1,1,2,3,2,1,888,3,3,3,1,1,2,888,2,2,1,888,888,2,3,3,3,1,3,1,2,888,2,2,3,2,888,3,3,3,1,888,3,1,888,2,888,1,1,888,1,3,1,3,2,3,1,2,3,1,1,3,1,3,3,1,2,3,2,1,2,3,888,3,2,2,3,1,2,3,2,2,2,3,3,1,3,1,3,1,2,1,2,3,3,2,1,1,1,1,888,1,3,888,1,888,1,3,888,2,2,1,3,2,2,1,2,3,3,888,3,1,2,3,3,1,888,888,888,888,3,3,888,888,3,1,888,1,1,2,888,3,2,3,888,1,3,1,1,3,1,2,1,2,3,888,3,3,1,2,3,1,3,3,2,3,1,2,3,2,3,888,888,2,1,3,1,1,2,1,3,3,1,3,3,2,2,1,3,3,1,2,1,1,1,888,2,2,888,3,3,1,2,3,2,888,1,3,1,2,3,1,1,888,2,3,1,888,888,3,2,888,2,888,1,3,888,888,888,888,888,888,888,3,3,3,2,2,1,2,1,1,3,1,3,3,2,1,3,3,3,888,1,1,888,2,2,888,1,3,888,888,1,1,3,1,888,3,2,3,3,1,3,3,1,3,1,2,2,888,3,888,1,3,888,2,2,2,1,2,3,2,2,1,1,2,3,3,3,2,1,1,3,1,3,3,2,1,2,2],["2014-01-02","2012-08-05","2013-07-19","2014-03-18","2014-07-01","2013-02-12",null,"2014-01-01","2012-09-07","2012-11-30","2014-03-12","2014-02-15","2012-10-28",null,"2013-05-20","2013-08-23","2013-09-23",null,null,"2013-02-12","2013-12-05","2013-02-15","2012-07-22","2013-02-02","2012-11-15","2013-03-30","2014-01-11",null,"2014-02-07","2014-01-25","2013-03-24","2013-08-29",null,"2014-05-22","2012-10-02","2013-01-05","2013-10-08",null,"2013-07-31","2013-05-30",null,"2013-02-04",null,"2014-03-12","2012-10-28",null,"2013-09-23","2013-03-19","2013-08-08","2013-10-26","2013-01-05","2013-07-26","2013-03-02","2013-10-25","2012-09-02","2013-01-25","2013-03-13","2013-02-20","2013-12-20","2013-10-17","2013-06-16","2013-03-16","2012-07-20","2013-05-13","2013-11-21","2012-09-12","2014-03-17","2013-09-22",null,"2012-12-12","2014-03-12","2013-01-13","2013-08-27","2014-02-21","2013-10-06","2013-09-27","2013-10-24","2014-01-22","2013-03-15","2013-01-23","2013-12-02","2013-10-02","2013-10-31","2012-09-19","2012-11-19","2013-03-21","2013-08-25","2012-08-30","2013-10-13","2013-07-08","2014-04-23","2013-12-09","2013-10-12","2012-12-07","2012-11-17","2014-05-11",null,"2013-07-05","2013-11-27",null,"2013-08-05",null,"2014-01-08","2013-09-16",null,"2014-01-17","2013-11-28","2012-12-26","2013-10-14","2013-12-16","2013-11-02","2013-03-10","2014-01-04","2013-05-13","2012-09-07",null,"2013-06-23","2013-12-31","2013-05-14","2012-09-15","2013-12-20","2013-10-28",null,null,null,null,"2013-12-20","2013-02-09",null,null,"2013-05-09","2012-10-22",null,"2014-02-08","2012-12-06","2013-12-29",null,"2013-02-09","2012-10-24","2013-09-21",null,"2013-05-07","2013-02-06","2013-09-10","2013-06-14","2013-01-25","2013-08-23","2012-12-07","2012-12-29","2013-04-20","2013-08-05",null,"2013-07-04","2013-04-12","2013-09-03","2013-12-26","2013-11-09","2013-10-08","2012-07-31","2012-12-01","2012-12-25","2014-01-18","2014-04-12","2013-10-25","2013-01-15","2013-08-02","2013-03-04",null,null,"2013-05-15","2013-01-26","2013-03-24","2013-07-19","2014-02-03","2013-06-19","2014-04-20","2013-04-05","2013-08-15","2012-12-23","2013-03-03","2014-01-14","2013-02-10","2013-09-27","2014-02-28","2013-06-03","2012-12-26","2013-01-01","2012-09-08","2013-11-17","2013-09-17","2013-07-22",null,"2013-10-11","2012-10-02",null,"2014-03-29","2012-11-30","2013-11-16","2012-11-10","2012-09-26","2013-12-28",null,"2013-06-13","2014-02-12","2012-09-27","2012-12-24","2012-12-15","2013-04-24","2013-02-27",null,"2012-11-11","2012-09-17","2013-10-23",null,null,"2012-10-29","2013-01-05",null,"2013-04-03",null,"2012-07-29","2013-04-03",null,null,null,null,null,null,null,"2013-01-27","2013-09-15","2014-03-30","2012-10-31","2013-05-31","2013-08-09","2013-05-27","2012-09-19","2014-01-27","2014-01-19","2014-04-17","2013-08-08","2013-04-26","2013-12-04","2013-02-26","2013-08-15","2013-02-16","2013-02-26",null,"2013-12-13","2012-11-18",null,"2013-02-17","2014-02-11",null,"2013-04-03","2013-07-06",null,null,"2012-07-09","2014-04-02","2013-12-28","2013-04-27",null,"2013-05-09","2013-06-08","2012-12-19","2014-03-20","2013-02-12","2013-02-01","2013-10-02","2013-04-05","2012-10-08","2014-09-02","2012-10-09","2013-02-20",null,"2013-04-08",null,"2013-08-28","2014-05-14",null,"2013-07-04","2012-12-14","2013-05-05","2014-01-22","2013-12-16","2014-01-14","2014-01-27","2013-01-22","2013-12-19","2014-01-11","2013-05-01","2013-09-01","2013-07-07","2012-09-19","2013-02-17","2013-05-19","2013-01-19","2013-09-16","2013-09-21","2013-09-21","2013-07-10","2013-02-01","2013-02-28","2013-04-26","2012-12-17"],["00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00",null,null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00",null,"00:00:00",null,"00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00",null,"00:00:00",null,"00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,null,null,null,"00:00:00","00:00:00",null,null,"00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00",null,null,"00:00:00","00:00:00",null,"00:00:00",null,"00:00:00","00:00:00",null,null,null,null,null,null,null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00",null,"00:00:00","00:00:00",null,"00:00:00","00:00:00",null,null,"00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00",null,"00:00:00",null,"00:00:00","00:00:00",null,"00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00","00:00:00"],["2014-01-02T00:00:00Z","2012-08-05T00:00:00Z","2013-07-19T00:00:00Z","2014-03-18T00:00:00Z","2014-07-01T00:00:00Z","2013-02-12T00:00:00Z",null,"2014-01-01T00:00:00Z","2012-09-07T00:00:00Z","2012-11-30T00:00:00Z","2014-03-12T00:00:00Z","2014-02-15T00:00:00Z","2012-10-28T00:00:00Z",null,"2013-05-20T00:00:00Z","2013-08-23T00:00:00Z","2013-09-23T00:00:00Z",null,null,"2013-02-12T00:00:00Z","2013-12-05T00:00:00Z","2013-02-15T00:00:00Z","2012-07-22T00:00:00Z","2013-02-02T00:00:00Z","2012-11-15T00:00:00Z","2013-03-30T00:00:00Z","2014-01-11T00:00:00Z",null,"2014-02-07T00:00:00Z","2014-01-25T00:00:00Z","2013-03-24T00:00:00Z","2013-08-29T00:00:00Z",null,"2014-05-22T00:00:00Z","2012-10-02T00:00:00Z","2013-01-05T00:00:00Z","2013-10-08T00:00:00Z",null,"2013-07-31T00:00:00Z","2013-05-30T00:00:00Z",null,"2013-02-04T00:00:00Z",null,"2014-03-12T00:00:00Z","2012-10-28T00:00:00Z",null,"2013-09-23T00:00:00Z","2013-03-19T00:00:00Z","2013-08-08T00:00:00Z","2013-10-26T00:00:00Z","2013-01-05T00:00:00Z","2013-07-26T00:00:00Z","2013-03-02T00:00:00Z","2013-10-25T00:00:00Z","2012-09-02T00:00:00Z","2013-01-25T00:00:00Z","2013-03-13T00:00:00Z","2013-02-20T00:00:00Z","2013-12-20T00:00:00Z","2013-10-17T00:00:00Z","2013-06-16T00:00:00Z","2013-03-16T00:00:00Z","2012-07-20T00:00:00Z","2013-05-13T00:00:00Z","2013-11-21T00:00:00Z","2012-09-12T00:00:00Z","2014-03-17T00:00:00Z","2013-09-22T00:00:00Z",null,"2012-12-12T00:00:00Z","2014-03-12T00:00:00Z","2013-01-13T00:00:00Z","2013-08-27T00:00:00Z","2014-02-21T00:00:00Z","2013-10-06T00:00:00Z","2013-09-27T00:00:00Z","2013-10-24T00:00:00Z","2014-01-22T00:00:00Z","2013-03-15T00:00:00Z","2013-01-23T00:00:00Z","2013-12-02T00:00:00Z","2013-10-02T00:00:00Z","2013-10-31T00:00:00Z","2012-09-19T00:00:00Z","2012-11-19T00:00:00Z","2013-03-21T00:00:00Z","2013-08-25T00:00:00Z","2012-08-30T00:00:00Z","2013-10-13T00:00:00Z","2013-07-08T00:00:00Z","2014-04-23T00:00:00Z","2013-12-09T00:00:00Z","2013-10-12T00:00:00Z","2012-12-07T00:00:00Z","2012-11-17T00:00:00Z","2014-05-11T00:00:00Z",null,"2013-07-05T00:00:00Z","2013-11-27T00:00:00Z",null,"2013-08-05T00:00:00Z",null,"2014-01-08T00:00:00Z","2013-09-16T00:00:00Z",null,"2014-01-17T00:00:00Z","2013-11-28T00:00:00Z","2012-12-26T00:00:00Z","2013-10-14T00:00:00Z","2013-12-16T00:00:00Z","2013-11-02T00:00:00Z","2013-03-10T00:00:00Z","2014-01-04T00:00:00Z","2013-05-13T00:00:00Z","2012-09-07T00:00:00Z",null,"2013-06-23T00:00:00Z","2013-12-31T00:00:00Z","2013-05-14T00:00:00Z","2012-09-15T00:00:00Z","2013-12-20T00:00:00Z","2013-10-28T00:00:00Z",null,null,null,null,"2013-12-20T00:00:00Z","2013-02-09T00:00:00Z",null,null,"2013-05-09T00:00:00Z","2012-10-22T00:00:00Z",null,"2014-02-08T00:00:00Z","2012-12-06T00:00:00Z","2013-12-29T00:00:00Z",null,"2013-02-09T00:00:00Z","2012-10-24T00:00:00Z","2013-09-21T00:00:00Z",null,"2013-05-07T00:00:00Z","2013-02-06T00:00:00Z","2013-09-10T00:00:00Z","2013-06-14T00:00:00Z","2013-01-25T00:00:00Z","2013-08-23T00:00:00Z","2012-12-07T00:00:00Z","2012-12-29T00:00:00Z","2013-04-20T00:00:00Z","2013-08-05T00:00:00Z",null,"2013-07-04T00:00:00Z","2013-04-12T00:00:00Z","2013-09-03T00:00:00Z","2013-12-26T00:00:00Z","2013-11-09T00:00:00Z","2013-10-08T00:00:00Z","2012-07-31T00:00:00Z","2012-12-01T00:00:00Z","2012-12-25T00:00:00Z","2014-01-18T00:00:00Z","2014-04-12T00:00:00Z","2013-10-25T00:00:00Z","2013-01-15T00:00:00Z","2013-08-02T00:00:00Z","2013-03-04T00:00:00Z",null,null,"2013-05-15T00:00:00Z","2013-01-26T00:00:00Z","2013-03-24T00:00:00Z","2013-07-19T00:00:00Z","2014-02-03T00:00:00Z","2013-06-19T00:00:00Z","2014-04-20T00:00:00Z","2013-04-05T00:00:00Z","2013-08-15T00:00:00Z","2012-12-23T00:00:00Z","2013-03-03T00:00:00Z","2014-01-14T00:00:00Z","2013-02-10T00:00:00Z","2013-09-27T00:00:00Z","2014-02-28T00:00:00Z","2013-06-03T00:00:00Z","2012-12-26T00:00:00Z","2013-01-01T00:00:00Z","2012-09-08T00:00:00Z","2013-11-17T00:00:00Z","2013-09-17T00:00:00Z","2013-07-22T00:00:00Z",null,"2013-10-11T00:00:00Z","2012-10-02T00:00:00Z",null,"2014-03-29T00:00:00Z","2012-11-30T00:00:00Z","2013-11-16T00:00:00Z","2012-11-10T00:00:00Z","2012-09-26T00:00:00Z","2013-12-28T00:00:00Z",null,"2013-06-13T00:00:00Z","2014-02-12T00:00:00Z","2012-09-27T00:00:00Z","2012-12-24T00:00:00Z","2012-12-15T00:00:00Z","2013-04-24T00:00:00Z","2013-02-27T00:00:00Z",null,"2012-11-11T00:00:00Z","2012-09-17T00:00:00Z","2013-10-23T00:00:00Z",null,null,"2012-10-29T00:00:00Z","2013-01-05T00:00:00Z",null,"2013-04-03T00:00:00Z",null,"2012-07-29T00:00:00Z","2013-04-03T00:00:00Z",null,null,null,null,null,null,null,"2013-01-27T00:00:00Z","2013-09-15T00:00:00Z","2014-03-30T00:00:00Z","2012-10-31T00:00:00Z","2013-05-31T00:00:00Z","2013-08-09T00:00:00Z","2013-05-27T00:00:00Z","2012-09-19T00:00:00Z","2014-01-27T00:00:00Z","2014-01-19T00:00:00Z","2014-04-17T00:00:00Z","2013-08-08T00:00:00Z","2013-04-26T00:00:00Z","2013-12-04T00:00:00Z","2013-02-26T00:00:00Z","2013-08-15T00:00:00Z","2013-02-16T00:00:00Z","2013-02-26T00:00:00Z",null,"2013-12-13T00:00:00Z","2012-11-18T00:00:00Z",null,"2013-02-17T00:00:00Z","2014-02-11T00:00:00Z",null,"2013-04-03T00:00:00Z","2013-07-06T00:00:00Z",null,null,"2012-07-09T00:00:00Z","2014-04-02T00:00:00Z","2013-12-28T00:00:00Z","2013-04-27T00:00:00Z",null,"2013-05-09T00:00:00Z","2013-06-08T00:00:00Z","2012-12-19T00:00:00Z","2014-03-20T00:00:00Z","2013-02-12T00:00:00Z","2013-02-01T00:00:00Z","2013-10-02T00:00:00Z","2013-04-05T00:00:00Z","2012-10-08T00:00:00Z","2014-09-02T00:00:00Z","2012-10-09T00:00:00Z","2013-02-20T00:00:00Z",null,"2013-04-08T00:00:00Z",null,"2013-08-28T00:00:00Z","2014-05-14T00:00:00Z",null,"2013-07-04T00:00:00Z","2012-12-14T00:00:00Z","2013-05-05T00:00:00Z","2014-01-22T00:00:00Z","2013-12-16T00:00:00Z","2014-01-14T00:00:00Z","2014-01-27T00:00:00Z","2013-01-22T00:00:00Z","2013-12-19T00:00:00Z","2014-01-11T00:00:00Z","2013-05-01T00:00:00Z","2013-09-01T00:00:00Z","2013-07-07T00:00:00Z","2012-09-19T00:00:00Z","2013-02-17T00:00:00Z","2013-05-19T00:00:00Z","2013-01-19T00:00:00Z","2013-09-16T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-07-10T00:00:00Z","2013-02-01T00:00:00Z","2013-02-28T00:00:00Z","2013-04-26T00:00:00Z","2012-12-17T00:00:00Z"],["H","H","H","H","H","H",null,"H","H","H","H","H","H",null,"H","H","H",null,null,"H","H","H","H","H","H","H","H",null,"H","H","H","H",null,"H","H","H","H",null,"H","H",null,"H",null,"H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H",null,"H",null,"H","H",null,"H","H","H","H","H","H","H","H","H","H",null,"H","H","H","H","H","H",null,null,null,null,"H","H",null,null,"H","H",null,"H","H","H",null,"H","H","H",null,"H","H","H","H","H","H","H","H","H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H",null,"H","H","H","H","H","H",null,"H","H","H","H","H","H","H",null,"H","H","H",null,null,"H","H",null,"H",null,"H","H",null,null,null,null,null,null,null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H",null,"H","H",null,"H","H",null,null,"H","H","H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H",null,"H",null,"H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H"],["2014-07-02","2012-09-01","2014-01-14","2014-03-31","2014-12-30","2013-03-09",null,"2014-07-09","2012-09-16","2013-01-23","2014-09-09","2014-08-16","2013-04-28",null,"2013-06-26","2014-02-20","2014-03-16",null,null,"2013-03-18","2013-12-09","2013-03-24","2013-01-20","2013-08-03","2013-01-12","2013-09-22","2014-07-10",null,"2014-05-31","2014-07-26","2013-06-14","2013-11-05",null,"2014-11-20","2013-04-02","2013-01-26","2014-03-18",null,"2013-08-05","2013-11-27",null,"2013-08-06",null,"2014-03-25","2013-04-28",null,"2014-03-24","2013-04-30","2014-02-05","2014-04-26","2013-02-12","2013-10-13","2013-08-31","2013-12-24","2012-12-04","2013-03-16","2013-09-14","2013-06-13","2013-12-26","2013-12-11","2013-06-29","2013-09-06","2013-01-11","2013-06-03","2014-04-19","2013-03-13","2014-05-07","2014-03-21",null,"2012-12-13","2014-09-11","2013-02-21","2013-09-25","2014-07-08","2013-11-18","2013-10-24","2013-12-22","2014-03-20","2013-06-17","2013-07-07","2014-02-01","2014-03-31","2014-05-16","2013-04-04","2013-05-27","2013-04-04","2013-10-09","2012-11-04","2013-12-06","2013-08-05","2014-07-04","2014-02-14","2014-04-18","2013-06-11","2013-01-09","2014-11-01",null,null,"2013-12-18",null,"2013-12-05",null,"2014-01-26","2013-09-28",null,"2014-07-25","2014-02-27","2013-06-24","2014-05-13","2013-12-30","2014-01-23","2013-09-08","2014-01-25",null,"2013-02-01",null,"2013-12-19","2014-07-28","2013-06-18","2012-09-24","2013-12-24","2014-04-26",null,null,null,null,"2014-01-01","2013-03-01",null,null,"2013-11-11","2013-04-28",null,"2014-03-21","2013-06-04","2014-04-06",null,"2013-02-22","2012-11-29","2013-09-21",null,"2013-11-05","2013-03-22","2014-03-08","2013-12-12","2013-05-03","2014-03-01","2013-06-05","2013-06-28","2013-06-18","2014-02-13",null,"2013-08-28","2013-04-19","2014-01-28","2014-07-02","2013-12-14","2014-04-08","2012-08-28","2013-06-01","2013-06-26","2014-04-27","2014-10-09","2014-04-25","2013-03-27","2013-09-26","2013-06-11",null,null,"2013-08-06","2013-06-13","2013-05-23","2014-01-17","2014-06-16","2013-12-19","2014-10-19","2013-10-02","2013-08-25","2013-06-24","2013-03-07","2014-01-18","2013-08-09","2013-10-29","2014-08-29","2013-08-13","2013-02-10","2013-07-05","2013-01-22","2014-05-18","2014-03-23","2013-08-01",null,"2013-11-13","2012-10-20",null,"2014-04-27","2013-03-19","2014-05-17","2013-05-12","2013-03-27","2014-06-28",null,"2013-10-11","2014-03-01","2012-11-21","2013-02-26","2013-02-15","2013-05-23","2013-07-06",null,"2013-05-01","2013-02-09","2014-04-24",null,null,"2013-02-18","2013-07-12",null,"2013-04-29",null,"2013-02-10","2013-05-30",null,null,null,null,null,null,null,"2013-02-05","2014-03-24","2014-10-05","2013-05-06","2013-07-01","2014-02-05","2013-11-22","2013-03-25","2014-07-28","2014-05-16","2014-10-16","2013-10-08","2013-10-23","2014-06-17","2013-09-07","2013-08-19","2013-08-18","2013-05-07",null,"2014-01-25","2013-05-27",null,"2013-03-05","2014-04-21",null,"2013-10-02","2013-07-07",null,null,"2013-01-20","2014-10-16","2014-01-02","2013-11-03",null,"2013-08-25","2013-08-01","2013-01-24","2014-09-20","2013-08-10","2013-05-11","2014-04-04","2013-10-11","2013-04-12","2015-03-05","2013-02-27","2013-03-31",null,"2013-06-28",null,"2013-10-07","2014-09-21",null,"2014-01-09","2013-02-27","2013-11-20","2014-07-22","2014-06-17","2014-07-16","2014-07-28","2013-07-24","2014-02-21","2014-03-14","2013-10-14","2014-03-03","2013-07-16","2012-10-31","2013-07-31","2013-11-17","2013-07-29","2013-10-12","2013-11-29","2014-01-31","2014-01-09","2013-04-18","2013-08-29","2013-08-01","2013-02-11"],["2014-07-02T23:59:59Z","2012-09-01T23:59:59Z","2014-01-14T23:59:59Z","2014-03-31T23:59:59Z","2014-12-30T23:59:59Z","2013-03-09T23:59:59Z",null,"2014-07-09T23:59:59Z","2012-09-16T23:59:59Z","2013-01-23T23:59:59Z","2014-09-09T23:59:59Z","2014-08-16T23:59:59Z","2013-04-28T23:59:59Z",null,"2013-06-26T23:59:59Z","2014-02-20T23:59:59Z","2014-03-16T23:59:59Z",null,null,"2013-03-18T23:59:59Z","2013-12-09T23:59:59Z","2013-03-24T23:59:59Z","2013-01-20T23:59:59Z","2013-08-03T23:59:59Z","2013-01-12T23:59:59Z","2013-09-22T23:59:59Z","2014-07-10T23:59:59Z",null,"2014-05-31T23:59:59Z","2014-07-26T23:59:59Z","2013-06-14T23:59:59Z","2013-11-05T23:59:59Z",null,"2014-11-20T23:59:59Z","2013-04-02T23:59:59Z","2013-01-26T23:59:59Z","2014-03-18T23:59:59Z",null,"2013-08-05T23:59:59Z","2013-11-27T23:59:59Z",null,"2013-08-06T23:59:59Z",null,"2014-03-25T23:59:59Z","2013-04-28T23:59:59Z",null,"2014-03-24T23:59:59Z","2013-04-30T23:59:59Z","2014-02-05T23:59:59Z","2014-04-26T23:59:59Z","2013-02-12T23:59:59Z","2013-10-13T23:59:59Z","2013-08-31T23:59:59Z","2013-12-24T23:59:59Z","2012-12-04T23:59:59Z","2013-03-16T23:59:59Z","2013-09-14T23:59:59Z","2013-06-13T23:59:59Z","2013-12-26T23:59:59Z","2013-12-11T23:59:59Z","2013-06-29T23:59:59Z","2013-09-06T23:59:59Z","2013-01-11T23:59:59Z","2013-06-03T23:59:59Z","2014-04-19T23:59:59Z","2013-03-13T23:59:59Z","2014-05-07T23:59:59Z","2014-03-21T23:59:59Z",null,"2012-12-13T23:59:59Z","2014-09-11T23:59:59Z","2013-02-21T23:59:59Z","2013-09-25T23:59:59Z","2014-07-08T23:59:59Z","2013-11-18T23:59:59Z","2013-10-24T23:59:59Z","2013-12-22T23:59:59Z","2014-03-20T23:59:59Z","2013-06-17T23:59:59Z","2013-07-07T23:59:59Z","2014-02-01T23:59:59Z","2014-03-31T23:59:59Z","2014-05-16T23:59:59Z","2013-04-04T23:59:59Z","2013-05-27T23:59:59Z","2013-04-04T23:59:59Z","2013-10-09T23:59:59Z","2012-11-04T23:59:59Z","2013-12-06T23:59:59Z","2013-08-05T23:59:59Z","2014-07-04T23:59:59Z","2014-02-14T23:59:59Z","2014-04-18T23:59:59Z","2013-06-11T23:59:59Z","2013-01-09T23:59:59Z","2014-11-01T23:59:59Z",null,null,"2013-12-18T23:59:59Z",null,"2013-12-05T23:59:59Z",null,"2014-01-26T23:59:59Z","2013-09-28T23:59:59Z",null,"2014-07-25T23:59:59Z","2014-02-27T23:59:59Z","2013-06-24T23:59:59Z","2014-05-13T23:59:59Z","2013-12-30T23:59:59Z","2014-01-23T23:59:59Z","2013-09-08T23:59:59Z","2014-01-25T23:59:59Z",null,"2013-02-01T23:59:59Z",null,"2013-12-19T23:59:59Z","2014-07-28T23:59:59Z","2013-06-18T23:59:59Z","2012-09-24T23:59:59Z","2013-12-24T23:59:59Z","2014-04-26T23:59:59Z",null,null,null,null,"2014-01-01T23:59:59Z","2013-03-01T23:59:59Z",null,null,"2013-11-11T23:59:59Z","2013-04-28T23:59:59Z",null,"2014-03-21T23:59:59Z","2013-06-04T23:59:59Z","2014-04-06T23:59:59Z",null,"2013-02-22T23:59:59Z","2012-11-29T23:59:59Z","2013-09-21T23:59:59Z",null,"2013-11-05T23:59:59Z","2013-03-22T23:59:59Z","2014-03-08T23:59:59Z","2013-12-12T23:59:59Z","2013-05-03T23:59:59Z","2014-03-01T23:59:59Z","2013-06-05T23:59:59Z","2013-06-28T23:59:59Z","2013-06-18T23:59:59Z","2014-02-13T23:59:59Z",null,"2013-08-28T23:59:59Z","2013-04-19T23:59:59Z","2014-01-28T23:59:59Z","2014-07-02T23:59:59Z","2013-12-14T23:59:59Z","2014-04-08T23:59:59Z","2012-08-28T23:59:59Z","2013-06-01T23:59:59Z","2013-06-26T23:59:59Z","2014-04-27T23:59:59Z","2014-10-09T23:59:59Z","2014-04-25T23:59:59Z","2013-03-27T23:59:59Z","2013-09-26T23:59:59Z","2013-06-11T23:59:59Z",null,null,"2013-08-06T23:59:59Z","2013-06-13T23:59:59Z","2013-05-23T23:59:59Z","2014-01-17T23:59:59Z","2014-06-16T23:59:59Z","2013-12-19T23:59:59Z","2014-10-19T23:59:59Z","2013-10-02T23:59:59Z","2013-08-25T23:59:59Z","2013-06-24T23:59:59Z","2013-03-07T23:59:59Z","2014-01-18T23:59:59Z","2013-08-09T23:59:59Z","2013-10-29T23:59:59Z","2014-08-29T23:59:59Z","2013-08-13T23:59:59Z","2013-02-10T23:59:59Z","2013-07-05T23:59:59Z","2013-01-22T23:59:59Z","2014-05-18T23:59:59Z","2014-03-23T23:59:59Z","2013-08-01T23:59:59Z",null,"2013-11-13T23:59:59Z","2012-10-20T23:59:59Z",null,"2014-04-27T23:59:59Z","2013-03-19T23:59:59Z","2014-05-17T23:59:59Z","2013-05-12T23:59:59Z","2013-03-27T23:59:59Z","2014-06-28T23:59:59Z",null,"2013-10-11T23:59:59Z","2014-03-01T23:59:59Z","2012-11-21T23:59:59Z","2013-02-26T23:59:59Z","2013-02-15T23:59:59Z","2013-05-23T23:59:59Z","2013-07-06T23:59:59Z",null,"2013-05-01T23:59:59Z","2013-02-09T23:59:59Z","2014-04-24T23:59:59Z",null,null,"2013-02-18T23:59:59Z","2013-07-12T23:59:59Z",null,"2013-04-29T23:59:59Z",null,"2013-02-10T23:59:59Z","2013-05-30T23:59:59Z",null,null,null,null,null,null,null,"2013-02-05T23:59:59Z","2014-03-24T23:59:59Z","2014-10-05T23:59:59Z","2013-05-06T23:59:59Z","2013-07-01T23:59:59Z","2014-02-05T23:59:59Z","2013-11-22T23:59:59Z","2013-03-25T23:59:59Z","2014-07-28T23:59:59Z","2014-05-16T23:59:59Z","2014-10-16T23:59:59Z","2013-10-08T23:59:59Z","2013-10-23T23:59:59Z","2014-06-17T23:59:59Z","2013-09-07T23:59:59Z","2013-08-19T23:59:59Z","2013-08-18T23:59:59Z","2013-05-07T23:59:59Z",null,"2014-01-25T23:59:59Z","2013-05-27T23:59:59Z",null,"2013-03-05T23:59:59Z","2014-04-21T23:59:59Z",null,"2013-10-02T23:59:59Z","2013-07-07T23:59:59Z",null,null,"2013-01-20T23:59:59Z","2014-10-16T23:59:59Z","2014-01-02T23:59:59Z","2013-11-03T23:59:59Z",null,"2013-08-25T23:59:59Z","2013-08-01T23:59:59Z","2013-01-24T23:59:59Z","2014-09-20T23:59:59Z","2013-08-10T23:59:59Z","2013-05-11T23:59:59Z","2014-04-04T23:59:59Z","2013-10-11T23:59:59Z","2013-04-12T23:59:59Z","2015-03-05T23:59:59Z","2013-02-27T23:59:59Z","2013-03-31T23:59:59Z",null,"2013-06-28T23:59:59Z",null,"2013-10-07T23:59:59Z","2014-09-21T23:59:59Z",null,"2014-01-09T23:59:59Z","2013-02-27T23:59:59Z","2013-11-20T23:59:59Z","2014-07-22T23:59:59Z","2014-06-17T23:59:59Z","2014-07-16T23:59:59Z","2014-07-28T23:59:59Z","2013-07-24T23:59:59Z","2014-02-21T23:59:59Z","2014-03-14T23:59:59Z","2013-10-14T23:59:59Z","2014-03-03T23:59:59Z","2013-07-16T23:59:59Z","2012-10-31T23:59:59Z","2013-07-31T23:59:59Z","2013-11-17T23:59:59Z","2013-07-29T23:59:59Z","2013-10-12T23:59:59Z","2013-11-29T23:59:59Z","2014-01-31T23:59:59Z","2014-01-09T23:59:59Z","2013-04-18T23:59:59Z","2013-08-29T23:59:59Z","2013-08-01T23:59:59Z","2013-02-11T23:59:59Z"],["H","H","H","H","H","H",null,"H","H","H","H","H","H",null,"H","H","H",null,null,"H","H","H","H","H","H","H","H",null,"H","H","H","H",null,"H","H","H","H",null,"H","H",null,"H",null,"H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,null,"H",null,"H",null,"H","H",null,"H","H","H","H","H","H","H","H",null,"H",null,"H","H","H","H","H","H",null,null,null,null,"H","H",null,null,"H","H",null,"H","H","H",null,"H","H","H",null,"H","H","H","H","H","H","H","H","H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H",null,"H","H","H","H","H","H",null,"H","H","H","H","H","H","H",null,"H","H","H",null,null,"H","H",null,"H",null,"H","H",null,null,null,null,null,null,null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H",null,"H","H",null,"H","H",null,"H","H",null,null,"H","H","H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H",null,"H",null,"H","H",null,"H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H","H"],[182,28,180,14,183,26,null,190,10,55,182,183,183,null,38,182,175,null,null,35,5,38,183,183,59,177,181,null,114,183,83,69,null,183,183,22,162,null,6,182,null,184,null,14,183,null,183,43,182,183,39,80,183,61,94,51,186,114,7,56,14,175,176,22,150,183,52,181,null,2,184,40,30,138,44,28,60,58,95,166,62,181,198,198,190,15,46,67,55,29,73,68,189,187,54,175,null,null,22,null,123,null,19,13,null,190,92,181,212,15,83,183,22,null,148,null,180,210,36,10,5,181,null,null,null,null,13,21,null,null,187,189,null,42,181,99,null,14,37,1,null,183,45,180,182,99,191,181,182,60,193,null,56,8,148,189,36,183,29,183,184,100,181,183,72,56,100,null,null,84,139,61,183,134,184,183,181,11,184,5,5,181,33,183,72,47,186,137,183,188,11,null,34,19,null,30,110,183,184,183,183,null,121,18,56,65,63,30,130,null,172,146,184,null,null,113,189,null,27,null,197,58,null,null,null,null,null,null,null,10,191,190,188,32,181,180,188,183,118,183,62,181,196,194,5,184,71,null,44,191,null,17,70,null,183,2,null,null,196,198,6,191,null,109,55,37,185,180,100,185,190,187,185,142,40,null,82,null,41,131,null,190,76,200,182,184,184,183,184,65,63,167,184,10,43,165,183,192,27,70,133,184,77,183,98,57],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-01-14",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-11-01",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-08-02",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-01-14",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-11-01",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-08-02",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,61,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,175,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,12,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,2,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"SUDDEN DEATH",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"COMPLETED SUICIDE",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"MYOCARDIAL INFARCTION",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"AE",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"AE",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"AE",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"ADVERSE EVENT",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"ADVERSE EVENT",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"ADVERSE EVENT",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,1,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,"2013-12-20",null,null,null,null,null,null,"2013-09-05",null,null,null,"2013-04-18","2012-09-30",null,null,null,null,null,null,null,null,"2013-09-22",null,null,null,null,"2013-12-08",null,null,null,null,"2014-04-01",null,null,"2013-09-18",null,"2013-12-28",null,null,"2012-11-14",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-02-09",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-09-01",null,null,"2013-12-02",null,"2014-02-23",null,null,"2013-02-26",null,null,null,null,null,null,null,null,null,null,"2013-01-14",null,null,null,null,null,null,"2012-10-12","2014-02-14","2013-07-25","2012-10-13",null,null,"2014-02-14","2013-02-07",null,null,"2014-01-23",null,null,null,"2013-01-14",null,null,null,"2012-08-13",null,null,null,null,null,null,null,null,null,null,"2013-09-16",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-11-18","2014-01-17",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-03-05",null,null,"2013-11-16",null,null,null,null,null,null,"2013-11-29",null,null,null,null,null,null,null,"2013-12-30",null,null,null,"2014-04-21","2013-06-02",null,null,"2013-12-13",null,"2014-03-17",null,null,"2012-11-08","2012-10-04","2013-10-24","2014-02-28","2014-03-27","2013-08-05","2013-12-20",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-04-17",null,null,"2013-01-22",null,null,"2013-06-03",null,null,"2012-10-24","2014-03-29",null,null,null,null,"2014-02-14",null,null,null,null,null,null,null,null,null,null,null,null,"2013-08-15",null,"2013-08-06",null,null,"2013-11-25",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],["2014-07-02","2012-09-02","2014-01-14","2014-04-14","2014-12-30","2013-03-29",null,"2014-07-09","2012-09-17","2013-01-23","2014-09-09","2014-08-16","2013-04-29",null,"2013-06-30","2014-02-20","2014-04-01",null,null,"2013-03-23","2013-12-12","2013-03-25","2013-01-20","2013-08-03","2013-01-14","2013-09-22","2014-07-11",null,"2014-06-14","2014-07-26","2013-06-14","2013-11-05",null,"2014-11-20","2013-04-02","2013-02-07","2014-03-18",null,"2013-08-14","2013-11-27",null,"2013-08-06",null,"2014-03-25","2013-04-28",null,"2014-03-24","2013-04-30","2014-02-05","2014-04-26","2013-02-13","2013-11-17","2013-08-31","2013-12-24","2012-12-24","2013-03-29","2013-09-14","2013-06-13","2013-12-31","2013-12-12","2013-07-22","2013-09-06","2013-01-21","2013-06-22","2014-05-19","2013-03-13","2014-05-24","2014-03-21",null,"2012-12-19","2014-09-11","2013-02-25","2013-10-02","2014-07-09","2013-11-24","2013-11-29","2013-12-24","2014-03-31","2013-06-20","2013-07-08","2014-02-03","2014-03-31","2014-05-16","2013-04-04","2013-05-27","2013-07-14","2013-10-31","2013-01-18","2013-12-06","2013-08-08","2014-08-06","2014-02-16","2014-04-18","2013-06-19","2013-01-12","2014-11-01",null,"2013-07-12","2014-05-11",null,"2013-12-05",null,"2014-02-07","2013-09-29",null,"2014-07-25","2014-03-17","2013-06-24","2014-05-14","2014-06-02","2014-01-23","2013-09-08","2014-03-07","2013-05-13","2013-02-20",null,"2013-12-19","2014-07-29","2013-06-25","2012-09-29","2014-01-08","2014-04-27",null,null,null,null,"2014-01-23","2013-03-09",null,null,"2013-11-11","2013-04-28",null,"2014-03-22","2013-06-04","2014-04-06",null,"2013-02-26","2012-12-06","2013-09-26",null,"2013-11-05","2013-04-04","2014-03-08","2013-12-12","2013-05-08","2014-03-01","2013-06-05","2013-06-28","2013-06-18","2014-02-13",null,"2013-09-10","2013-05-10","2014-01-28","2014-07-02","2014-01-31","2014-04-08","2012-09-01","2013-06-01","2013-06-26","2014-05-10","2014-10-09","2014-04-25","2013-03-31","2013-09-27","2013-06-25",null,null,"2013-08-07","2013-06-13","2013-05-31","2014-01-17","2014-06-25","2013-12-19","2014-10-19","2013-10-02","2013-08-29","2013-06-24","2013-03-08","2014-01-18","2013-08-10","2013-11-12","2014-08-29","2013-08-17","2013-02-15","2013-07-05","2013-02-23","2014-05-18","2014-03-23","2013-08-02",null,"2013-11-13","2012-10-23",null,"2014-05-09","2013-03-30","2014-05-17","2013-05-12","2013-03-27","2014-06-28",null,"2013-10-12","2014-03-01","2012-12-02","2013-03-03","2013-02-19","2013-05-25","2013-07-16",null,"2013-05-01","2013-03-12","2014-04-24",null,null,"2013-02-18","2013-07-12",null,"2013-05-02",null,"2013-02-10","2013-06-01",null,null,null,null,null,null,null,"2013-02-06","2014-03-24","2014-10-05","2013-05-06","2013-08-06","2014-02-05","2013-11-22","2013-03-25","2014-07-28","2014-05-17","2014-10-16","2013-10-16","2013-10-23","2014-06-17","2013-09-07","2013-08-22","2013-08-18","2013-05-13",null,"2014-04-26","2013-06-01",null,"2013-03-18","2014-04-22",null,"2013-10-03","2013-07-13",null,null,"2013-01-20","2014-10-16","2014-01-10","2013-11-03",null,"2013-08-28","2013-08-01","2013-01-26","2014-09-20","2013-08-10","2013-06-07","2014-04-05","2013-10-11","2013-04-13","2015-03-05","2013-03-03","2013-04-02",null,"2013-07-06",null,"2013-10-07","2014-09-30",null,"2014-01-09","2013-02-28","2013-11-20","2014-07-22","2014-06-17","2014-07-16","2014-07-28","2013-07-24","2014-02-21","2014-03-14","2013-10-20","2014-03-04","2013-07-30","2012-11-06","2013-08-02","2013-11-17","2013-07-31","2013-11-03","2013-11-29","2014-02-08","2014-01-09","2013-05-01","2013-08-29","2013-08-08","2013-02-18"],["COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED",null,"COMPLETED","DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED",null,"DISCONTINUED","COMPLETED","COMPLETED",null,null,"DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED",null,"DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED",null,"COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED",null,"DISCONTINUED","COMPLETED",null,"COMPLETED",null,"DISCONTINUED","COMPLETED",null,"COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED",null,"DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED",null,"DISCONTINUED","DISCONTINUED",null,"DISCONTINUED",null,"DISCONTINUED","DISCONTINUED",null,"COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED",null,"COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED",null,null,null,null,"DISCONTINUED","DISCONTINUED",null,null,"COMPLETED","COMPLETED",null,"DISCONTINUED","COMPLETED","DISCONTINUED",null,"DISCONTINUED","DISCONTINUED","DISCONTINUED",null,"COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED",null,"DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED",null,null,"DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED",null,"DISCONTINUED","DISCONTINUED",null,"DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","COMPLETED",null,"DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED","DISCONTINUED",null,"COMPLETED","DISCONTINUED","COMPLETED",null,null,"DISCONTINUED","COMPLETED",null,"DISCONTINUED",null,"COMPLETED","DISCONTINUED",null,null,null,null,null,null,null,"DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED",null,"DISCONTINUED","COMPLETED",null,"DISCONTINUED","DISCONTINUED",null,"COMPLETED","DISCONTINUED",null,null,"COMPLETED","COMPLETED","DISCONTINUED","COMPLETED",null,"DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED",null,"DISCONTINUED",null,"DISCONTINUED","DISCONTINUED",null,"COMPLETED","DISCONTINUED","COMPLETED","COMPLETED","COMPLETED","COMPLETED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","COMPLETED","DISCONTINUED","DISCONTINUED","DISCONTINUED","COMPLETED","DISCONTINUED","COMPLETED","DISCONTINUED","DISCONTINUED"],[null,"2013-02-18",null,"2014-09-15",null,"2013-07-28",null,null,"2013-02-22","2013-05-20",null,null,null,null,null,null,null,null,null,null,"2014-05-23","2013-08-04",null,null,null,null,null,null,null,null,"2013-10-08","2014-02-13",null,null,null,null,null,null,"2014-02-11",null,null,null,null,"2014-08-27",null,null,null,null,null,null,"2013-06-20",null,null,null,"2013-03-26",null,null,"2013-08-24",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-12-25",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-02-06",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-11-26","2013-06-22",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-09-18",null,null,"2013-09-22",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-07-27",null,null,null,null,null,null,null,"2013-08-14",null,null,null,null,null,null,null,null,"2014-01-11",null,null,null,null,null,null,null,"2013-11-17","2013-12-16","2013-06-22",null,null,"2013-08-13",null,null,null,null,"2013-04-08",null,null,"2013-10-07",null,null,"2014-11-15",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-03-05",null,null,null,"2014-03-15","2014-03-21","2014-03-08",null,"2013-07-24",null,null,"2013-06-03"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>COUNTRY<\/th>\n      <th>REGION1<\/th>\n      <th>REGION1N<\/th>\n      <th>AGE<\/th>\n      <th>AGEU<\/th>\n      <th>AGEGR1<\/th>\n      <th>AGEGR1N<\/th>\n      <th>SEX<\/th>\n      <th>RACE<\/th>\n      <th>ITTFL<\/th>\n      <th>RACEN<\/th>\n      <th>RACEGR1<\/th>\n      <th>RACEGR1N<\/th>\n      <th>ETHNIC<\/th>\n      <th>SAFFL<\/th>\n      <th>RANDFL<\/th>\n      <th>RANDDT<\/th>\n      <th>ARM<\/th>\n      <th>ARMCD<\/th>\n      <th>ACTARM<\/th>\n      <th>ACTARMCD<\/th>\n      <th>TRT01P<\/th>\n      <th>TRT01PN<\/th>\n      <th>TRT01A<\/th>\n      <th>TRT01AN<\/th>\n      <th>TRTSDT<\/th>\n      <th>TRTSTM<\/th>\n      <th>TRTSDTM<\/th>\n      <th>TRTSTMF<\/th>\n      <th>TRTEDT<\/th>\n      <th>TRTEDTM<\/th>\n      <th>TRTETMF<\/th>\n      <th>TRTDURD<\/th>\n      <th>DTHDTC<\/th>\n      <th>DTHDT<\/th>\n      <th>DTHDTF<\/th>\n      <th>DTHFL<\/th>\n      <th>DTHADY<\/th>\n      <th>LDDTHELD<\/th>\n      <th>DTHCAUS<\/th>\n      <th>DTHDOM<\/th>\n      <th>DTHCGR1<\/th>\n      <th>DTHCGR1N<\/th>\n      <th>SCRFDT<\/th>\n      <th>EOSDT<\/th>\n      <th>EOSSTT<\/th>\n      <th>FRVDT<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[7,8,11,15,17,27,29,37,42,43,47]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"COUNTRY","targets":5},{"name":"REGION1","targets":6},{"name":"REGION1N","targets":7},{"name":"AGE","targets":8},{"name":"AGEU","targets":9},{"name":"AGEGR1","targets":10},{"name":"AGEGR1N","targets":11},{"name":"SEX","targets":12},{"name":"RACE","targets":13},{"name":"ITTFL","targets":14},{"name":"RACEN","targets":15},{"name":"RACEGR1","targets":16},{"name":"RACEGR1N","targets":17},{"name":"ETHNIC","targets":18},{"name":"SAFFL","targets":19},{"name":"RANDFL","targets":20},{"name":"RANDDT","targets":21},{"name":"ARM","targets":22},{"name":"ARMCD","targets":23},{"name":"ACTARM","targets":24},{"name":"ACTARMCD","targets":25},{"name":"TRT01P","targets":26},{"name":"TRT01PN","targets":27},{"name":"TRT01A","targets":28},{"name":"TRT01AN","targets":29},{"name":"TRTSDT","targets":30},{"name":"TRTSTM","targets":31},{"name":"TRTSDTM","targets":32},{"name":"TRTSTMF","targets":33},{"name":"TRTEDT","targets":34},{"name":"TRTEDTM","targets":35},{"name":"TRTETMF","targets":36},{"name":"TRTDURD","targets":37},{"name":"DTHDTC","targets":38},{"name":"DTHDT","targets":39},{"name":"DTHDTF","targets":40},{"name":"DTHFL","targets":41},{"name":"DTHADY","targets":42},{"name":"LDDTHELD","targets":43},{"name":"DTHCAUS","targets":44},{"name":"DTHDOM","targets":45},{"name":"DTHCGR1","targets":46},{"name":"DTHCGR1N","targets":47},{"name":"SCRFDT","targets":48},{"name":"EOSDT","targets":49},{"name":"EOSSTT","targets":50},{"name":"FRVDT","targets":51}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
</section></section><section id="adae" class="level2"><h2 class="anchored" data-anchor-id="adae">ADAE</h2>
<p>임상시험에서의 이상반응(<strong>Adverse Events, AE</strong>) 데이터를 분석하기 위한 ADAE 데이터셋 생성 과정 <br><code>ADSL</code> + <code>AE</code> = <code>ADAE</code>라 생각하자.</p>
<p><strong>1. 개요</strong><br>
∙ 목적: 이상반응 데이터의 표준화된 분석을 위한 데이터셋 생성 <br><br> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;안전성 평가(Safety Analysis)의 핵심 데이터셋으로, 다음 목적에 사용:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ 이상반응 발생률 계산<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ 치료군 간 AE 비교<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ 중대한 이상반응(SAE) 추적<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ 규제 기관(FDA, PMDA 등) 제출용 <br><br> ∙ 입력 데이터:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ AE (기본 이상반응 데이터)<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ ADSL (환자 기본 정보) <br><br> ∙ 출력 데이터:<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;∘ ADAE (Analysis Dataset for Adverse Events)</p>
<p><strong>2. 주요변수</strong></p>
<div class="cell">
<div class="cell-output-display">
<table class="table table-striped table-hover table-condensed caption-top table-sm small">
<caption>식별자 변수</caption>
<thead><tr class="header">
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">변수명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">설명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">예시</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">필수여부</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">STUDYID</td>
<td style="text-align: left;">연구 식별자</td>
<td style="text-align: left;">"CDISCPILOT01"</td>
<td style="text-align: left;">Required</td>
</tr>
<tr class="even">
<td style="text-align: left;">USUBJID</td>
<td style="text-align: left;">환자 고유 ID</td>
<td style="text-align: left;">"01-701-1015"</td>
<td style="text-align: left;">Required</td>
</tr>
<tr class="odd">
<td style="text-align: left;">AESEQ</td>
<td style="text-align: left;">AE 시퀀스 번호</td>
<td style="text-align: left;">1, 2, 3...</td>
<td style="text-align: left;">Required</td>
</tr>
</tbody>
</table>
</div>
<div class="cell-output-display">
<table class="table table-striped table-hover table-condensed caption-top table-sm small">
<caption>시간 관련 변수</caption>
<thead><tr class="header">
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">변수명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">설명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">예시</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">AESTDT</td>
<td style="text-align: left;">AE 시작일</td>
<td style="text-align: left;">2023-05-20</td>
</tr>
<tr class="even">
<td style="text-align: left;">AEENDT</td>
<td style="text-align: left;">AE 종료일</td>
<td style="text-align: left;">2023-05-25</td>
</tr>
<tr class="odd">
<td style="text-align: left;">AEDUR</td>
<td style="text-align: left;">AE 지속일수 (파생 변수)</td>
<td style="text-align: left;">6</td>
</tr>
</tbody>
</table>
</div>
<div class="cell-output-display">
<table class="table table-striped table-hover table-condensed caption-top table-sm small">
<caption>이상반응 특성</caption>
<thead><tr class="header">
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">변수명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">설명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">예시.값</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">AETERM</td>
<td style="text-align: left;">보고된 AE 명칭</td>
<td style="text-align: left;">"Headache"</td>
</tr>
<tr class="even">
<td style="text-align: left;">AEDECOD</td>
<td style="text-align: left;">표준화된 AE 용어 (PT)</td>
<td style="text-align: left;">"HEADACHE"</td>
</tr>
<tr class="odd">
<td style="text-align: left;">AEBODSYS</td>
<td style="text-align: left;">신체계 (SOC)</td>
<td style="text-align: left;">"NERVOUS SYSTEM"</td>
</tr>
<tr class="even">
<td style="text-align: left;">AESEV</td>
<td style="text-align: left;">중증도</td>
<td style="text-align: left;">"MILD", "SEVERE"</td>
</tr>
</tbody>
</table>
</div>
<div class="cell-output-display">
<table class="table table-striped table-hover table-condensed caption-top table-sm small">
<caption>평가 변수</caption>
<thead><tr class="header">
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">변수명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">설명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">값.범위</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">AEREL</td>
<td style="text-align: left;">약물 관련성</td>
<td style="text-align: left;">"Y"/"N"/NA</td>
</tr>
<tr class="even">
<td style="text-align: left;">AESER</td>
<td style="text-align: left;">중대한 AE 여부</td>
<td style="text-align: left;">"Y"/"N"</td>
</tr>
<tr class="odd">
<td style="text-align: left;">AEOUT</td>
<td style="text-align: left;">결과</td>
<td style="text-align: left;">"RECOVERED"</td>
</tr>
</tbody>
</table>
</div>
<div class="cell-output-display">
<table class="table table-striped table-hover table-condensed caption-top table-sm small">
<caption>파생 플래그</caption>
<thead><tr class="header">
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">변수명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">설명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">생성.로직</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">TRTEMFL</td>
<td style="text-align: left;">치료 기간 중 발생 여부</td>
<td style="text-align: left;">AESTDT ≤ TRTEDT</td>
</tr>
<tr class="even">
<td style="text-align: left;">AESERFL</td>
<td style="text-align: left;">중대한 AE 플래그</td>
<td style="text-align: left;">AESER == 'Y'</td>
</tr>
<tr class="odd">
<td style="text-align: left;">AERELFL</td>
<td style="text-align: left;">관련성 플래그</td>
<td style="text-align: left;">AEREL == 'Y'</td>
</tr>
</tbody>
</table>
</div>
<div class="cell-output-display">
<table class="table table-striped table-hover table-condensed caption-top table-sm small">
<caption>치료 정보</caption>
<thead><tr class="header">
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">변수명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">설명</th>
<th data-quarto-table-cell-role="th" style="text-align: left; font-weight: bold; background-color: rgba(248, 249, 250, 255) !important;">출처</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">TRT01A</td>
<td style="text-align: left;">실제 치료군</td>
<td style="text-align: left;">ADSL</td>
</tr>
<tr class="even">
<td style="text-align: left;">TRTSDT</td>
<td style="text-align: left;">치료 시작일</td>
<td style="text-align: left;">ADSL</td>
</tr>
</tbody>
</table>
</div>
</div>
<section id="adae_example" class="level3"><h3 class="anchored" data-anchor-id="adae_example">ADAE_example</h3>
<p>예제는 아래와 같은 순서로 진행한다.</p>
<ul>
<li>data/Packages loading</li>
<li>metacore specification loading</li>
<li>ADSL variable selection</li>
<li>Derivation Building</li>
<li>Derive Analysis Dates</li>
<li>Derive Duration/Date of last dose</li>
<li>Derive Analysis/Occurence Flags</li>
<li>Derive Query Variables</li>
<li>Add ADSL variables</li>
<li>Applying metadate</li>
</ul>
<p><strong>Data/Packages loading</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb26" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://atorus-research.github.io/metacore/">metacore</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/pharmaverse/metatools">metatools</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://pharmaverse.github.io/pharmaversesdtm/">pharmaversesdtm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://pharmaverse.github.io/pharmaverseadam/">pharmaverseadam</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://pharmaverse.github.io/admiral/">admiral</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://atorus-research.github.io/xportr/">xportr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://dplyr.tidyverse.org">dplyr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://lubridate.tidyverse.org">lubridate</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://stringr.tidyverse.org">stringr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://glin.github.io/reactable/">reactable</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Read in input data</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaverseadam</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pharmaversesdtm</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># When SAS datasets are imported into R using haven::read_sas(), missing</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">convert_blanks_to_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><strong>Metacore specification loading</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb27" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 메타데이터 Excel 파일에서 사양 추출</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">spec_to_metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>  path <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"./safety_specs.xlsx"</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 메타데이터 파일 경로</span></span>
<span>  where_sep_sheet <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 모든 데이터셋이 한 시트에 있는 경우</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select_dataset</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADAE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ADAE 데이터셋 선택</span></span></code></pre></div></div>
</div>
<p><strong>ADSL variable selection</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb28" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ADSL에서 필요한 변수 선택</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_vars</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDT</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTEDT</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">DTHDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 시작/종료일, 사망일</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 데이터에 ADSL 변수 병합</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span>,          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ADSL 데이터</span></span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_vars</span>,        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 추가할 변수</span></span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 병합 키</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><strong>Derive Analysis Dates</strong> <br><br><strong>1. Analysis Date/Relative Analysis Day</strong> <br><code>admiral::derive_vars_dt()</code> and <code>admiral::derive_vars_dy()</code> <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb29" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 종료일(AENDT) 파생 (부분일자 대체)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AEN"</span>,           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 생성 변수: AENDT, AENDTF</span></span>
<span>      dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AEENDTC</span>,                     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 원본 AE 종료일 문자형</span></span>
<span>      date_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"last"</span>,          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 결측일은 월의 마지막 날로 대체 ("2023-02" → 2023-02-28)</span></span>
<span>      highest_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"M"</span>,          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 월까지만 있는 경우 대체 수행</span></span>
<span>      flag_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"auto"</span>           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 대체 여부 플래그 자동 생성 (AENDTF)</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 시작일(ASTDT) 파생 (추가 제약조건 적용)</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AST"</span>,           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 생성 변수: ASTDT, ASTDTF</span></span>
<span>      dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AESTDTC</span>,</span>
<span>      highest_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"M"</span>,</span>
<span>      flag_imputation <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"auto"</span>,</span>
<span>      min_dates <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,         <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료시작일보다 이전일 수 없음</span></span>
<span>      max_dates <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AENDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 종료일보다 이후일 수 없음</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 시작일 기준 상대일자 파생</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dy</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      reference_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDT</span>,           <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 기준일자</span></span>
<span>      source_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ASTDT</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AENDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 계산 대상</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>                                    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 생성 변수: ASTDY, AENDY</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AESTDTC"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASTDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASTDTF"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASTDY"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AEENDTC"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AENDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AENDTF"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AENDY"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-05670a7e3ac21a008854" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-05670a7e3ac21a008854">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1023"],["2014-01-03","2014-01-03","2014-01-09","2012-08-26","2012-08-07","2012-08-07"],["2014-01-03","2014-01-03","2014-01-09","2012-08-26","2012-08-07","2012-08-07"],[null,null,null,null,null,null],[2,2,8,22,3,3],[null,null,"2014-01-11",null,"2012-08-30",null],[null,null,"2014-01-11",null,"2012-08-30",null],[null,null,null,null,null,null],[null,null,10,null,26,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>USUBJID<\/th>\n      <th>AESTDTC<\/th>\n      <th>ASTDT<\/th>\n      <th>ASTDTF<\/th>\n      <th>ASTDY<\/th>\n      <th>AEENDTC<\/th>\n      <th>AENDT<\/th>\n      <th>AENDTF<\/th>\n      <th>AENDY<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[5,9]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"USUBJID","targets":1},{"name":"AESTDTC","targets":2},{"name":"ASTDT","targets":3},{"name":"ASTDTF","targets":4},{"name":"ASTDY","targets":5},{"name":"AEENDTC","targets":6},{"name":"AENDT","targets":7},{"name":"AENDTF","targets":8},{"name":"AENDY","targets":9}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><strong>2. AE duration</strong> <br><code>admiral::derive_vars_duration()</code> <br><code>admiral::derive_vars_joined()</code> : 다른 data(예: ex)로 부터 join할 수 있다. <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb30" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_duration</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    new_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ADURN</span>,        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 지속일수 변수명</span></span>
<span>    new_var_unit <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ADURU</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 단위 변수명 (기본값 "DAYS")</span></span>
<span>    start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ASTDT</span>,     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 시작일</span></span>
<span>    end_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AENDT</span>,       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 종료일</span></span>
<span>    add_one <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 시작일 포함 계산 (기본값 TRUE)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># EX 데이터 전처리</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># &gt; last dose date을 위해 ex에서 exposure information을 불러오자.</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dtc <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDTC</span>,</span>
<span>    new_vars_prefix <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EXEN"</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># EXENDT 생성</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># ADAE에 최종 투여일 병합</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_joined</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span>,</span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 기본 조인 키</span></span>
<span>    order <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,              <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># EXENDT 기준 정렬</span></span>
<span>    new_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>LDOSEDT <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 생성 변수</span></span>
<span>    join_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,          <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 조인 조건에 사용할 변수</span></span>
<span>    join_type <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"all"</span>,</span>
<span>    filter_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXDOSE</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">str_detect</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXTRT</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PLACEBO"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&amp;</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDT</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    filter_join <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">EXENDT</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ASTDT</span>,      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 시작일 이전의 투여기록만</span></span>
<span>    mode <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"last"</span>                       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 가장 가까운 기록 선택</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EXTRT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EXDOSE"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"EXENDT"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">datatable</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-bf54c9b9405071d0472a" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-bf54c9b9405071d0472a">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1028"],["PLACEBO","PLACEBO","PLACEBO","PLACEBO","PLACEBO","XANOMELINE"],[0,0,0,0,0,54],["2014-01-16","2014-06-18","2014-07-02","2012-08-27","2012-09-01","2013-08-01"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>USUBJID<\/th>\n      <th>EXTRT<\/th>\n      <th>EXDOSE<\/th>\n      <th>EXENDT<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":3},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"USUBJID","targets":1},{"name":"EXTRT","targets":2},{"name":"EXDOSE","targets":3},{"name":"EXENDT","targets":4}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb31" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASTDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AENDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADURN"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADURU"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"LDOSEDT"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-5808abab652af4d30fac" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-5808abab652af4d30fac">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1023"],["2014-01-03","2014-01-03","2014-01-09","2012-08-26","2012-08-07","2012-08-07"],[null,null,"2014-01-11",null,"2012-08-30",null],[null,null,3,null,24,null],[null,null,"DAYS",null,"DAYS",null],[null,null,null,null,null,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>USUBJID<\/th>\n      <th>ASTDT<\/th>\n      <th>AENDT<\/th>\n      <th>ADURN<\/th>\n      <th>ADURU<\/th>\n      <th>LDOSEDT<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":4},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"USUBJID","targets":1},{"name":"ASTDT","targets":2},{"name":"AENDT","targets":3},{"name":"ADURN","targets":4},{"name":"ADURU","targets":5},{"name":"LDOSEDT","targets":6}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><strong>Derive Flags</strong></p>
<p><code>admiral::derive_var_trtemfl()</code> : Treatment Emergent Flag (<code>TRTEMFL</code>) <br> &nbsp;&nbsp;&nbsp;&nbsp; # TRTSDT &lt; ASTDT, &nbsp;&nbsp; TRTEMFL: Y <br><code>admiral::derive_var_ontrtfl()</code> : On-Treatment Flag (<code>ONTRTFL</code>) <br> &nbsp;&nbsp;&nbsp;&nbsp; # TRTSDT &lt; ASTDT &lt; TRTEDT +30day, &nbsp;&nbsp; ONTRTFL: Y <br><code>admiral::derive_var_extreme_flag()</code> : AE occurance initial Flag (<code>AOCCIFL</code>) <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb32" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Derive TRTEMFL and ONTRTFL</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_var_trtemfl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ASTDT</span>,       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 시작일</span></span>
<span>    end_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AENDT</span>,         <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 종료일</span></span>
<span>    trt_start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDT</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 시작일</span></span>
<span>    trt_end_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTEDT</span>     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 종료일</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_var_ontrtfl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ASTDT</span>,        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AE 시작일</span></span>
<span>    ref_start_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTSDT</span>,   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 시작일</span></span>
<span>    ref_end_date <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TRTEDT</span>,     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 종료일</span></span>
<span>    ref_end_window <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span>        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료 종료 후 추가 기간 (일)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASTDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AENDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTSDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTEDT"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"TRTEMFL"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ONTRTFL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-d421b49a38dd52e2c7ef" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-d421b49a38dd52e2c7ef">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1023"],["2014-01-03","2014-01-03","2014-01-09","2012-08-26","2012-08-07","2012-08-07"],[null,null,"2014-01-11",null,"2012-08-30",null],["2014-01-02","2014-01-02","2014-01-02","2012-08-05","2012-08-05","2012-08-05"],["2014-07-02","2014-07-02","2014-07-02","2012-09-01","2012-09-01","2012-09-01"],["Y","Y","Y","Y","Y","Y"],["Y","Y","Y","Y","Y","Y"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>USUBJID<\/th>\n      <th>ASTDT<\/th>\n      <th>AENDT<\/th>\n      <th>TRTSDT<\/th>\n      <th>TRTEDT<\/th>\n      <th>TRTEMFL<\/th>\n      <th>ONTRTFL<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"USUBJID","targets":1},{"name":"ASTDT","targets":2},{"name":"AENDT","targets":3},{"name":"TRTSDT","targets":4},{"name":"TRTEDT","targets":5},{"name":"TRTEMFL","targets":6},{"name":"ONTRTFL","targets":7}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb33" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#Derive AOCCIFL</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># create temporary numeric ASEVN for sorting purpose</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">mutate</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>TEMP_AESEVN <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/integer.html">as.integer</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AESEV</span>, levels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SEVERE"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MODERATE"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MILD"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_var_extreme_flag</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    new_var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AOCCIFL</span>,                  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 새 플래그 변수명</span></span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 그룹화 기준 (환자별)</span></span>
<span>    order <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">TEMP_AESEVN</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ASTDT</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AESEQ</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 정렬 순서</span></span>
<span>    mode <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"first"</span>                      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 첫 번째 레코드 선택</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ASTDT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AESEQ"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AESEV"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AOCCIFL"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-40c71076c34fbc337242" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-40c71076c34fbc337242">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1023"],["2014-01-03","2014-01-03","2014-01-09","2012-08-07","2012-08-07","2012-08-07"],[1,2,3,2,1,4],["MILD","MILD","MILD","MODERATE","MILD","MILD"],["Y",null,null,"Y",null,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>USUBJID<\/th>\n      <th>ASTDT<\/th>\n      <th>AESEQ<\/th>\n      <th>AESEV<\/th>\n      <th>AOCCIFL<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":3},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"USUBJID","targets":1},{"name":"ASTDT","targets":2},{"name":"AESEQ","targets":3},{"name":"AESEV","targets":4},{"name":"AOCCIFL","targets":5}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><br><strong>Derive Query Variables</strong></p>
<p><strong>Query dataset</strong>: AE 관련 용어들을 정리해놓은 basket으로 mapping 규칙을 정의. ex) SMQs, CQs. <br><a href="https://pharmaverse.github.io/admiral/articles/queries_dataset.html/"><code>{admiral}</code>Queries Dataset Vignette</a> 참고. <br><br><code>admiral::derive_vars_query()</code>을 이용한다. <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb34" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">queries</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">admiral</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">queries</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/filter.html">filter</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">PREFIX</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"CQ01"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SMQ02"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># CQ01(커스텀)과 SMQ02(표준)만 필터링</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">queries</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">head</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">datatable</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-9c7843d3f137adc0e9e5" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-9c7843d3f137adc0e9e5">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["CQ01","CQ01","CQ01","CQ01","CQ01","SMQ02"],["Dermatologic events","Dermatologic events","Dermatologic events","Dermatologic events","Dermatologic events","Immune-Mediated Hypothyroidism"],[null,null,null,null,null,20000160],[null,null,null,null,null,"BROAD"],[null,null,null,null,null,1],["AELLT","AELLT","AELLT","AELLT","AELLT","AEDECOD"],["APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","ERYTHEMA","LOCALIZED ERYTHEMA","GENERALIZED PRURITUS","BIOPSY THYROID GLAND ABNORMAL"],[null,null,null,null,null,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>PREFIX<\/th>\n      <th>GRPNAME<\/th>\n      <th>GRPID<\/th>\n      <th>SCOPE<\/th>\n      <th>SCOPEN<\/th>\n      <th>SRCVAR<\/th>\n      <th>TERMCHAR<\/th>\n      <th>TERMNUM<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[3,5,8]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"PREFIX","targets":1},{"name":"GRPNAME","targets":2},{"name":"GRPID","targets":3},{"name":"SCOPE","targets":4},{"name":"SCOPEN","targets":5},{"name":"SRCVAR","targets":6},{"name":"TERMCHAR","targets":7},{"name":"TERMNUM","targets":8}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb35" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># query변수 생성</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_query</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>dataset_queries <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">queries</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/head.html">head</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AEDECOD"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"CQ01NAM"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SMQ02NAM"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-a0ba81d76b9296f931b0" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-a0ba81d76b9296f931b0">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1023"],["APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","DIARRHOEA","ERYTHEMA","ERYTHEMA","ERYTHEMA"],[null,null,null,"Dermatologic events","Dermatologic events","Dermatologic events"],[null,null,null,null,null,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>USUBJID<\/th>\n      <th>AEDECOD<\/th>\n      <th>CQ01NAM<\/th>\n      <th>SMQ02NAM<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"USUBJID","targets":1},{"name":"AEDECOD","targets":2},{"name":"CQ01NAM","targets":3},{"name":"SMQ02NAM","targets":4}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
<p><strong>Add ADSL variables</strong></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb36" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">derive_vars_merged</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset_add <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">select</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">negate_vars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl_vars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># adsl_vars에 없는 변수만 선택</span></span>
<span>    by_vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exprs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">STUDYID</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">USUBJID</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 키 변수</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><strong>Associated check perform, Create eSub XPT</strong> <br> - <a href="https://github.com/pharmaverse/metatools">metatools</a> &amp; <a href="https://atorus-research.github.io/xportr/">xportr</a> package 이용.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb37" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dir</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/tempfile.html">tempdir</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 임시 디렉토리 지정</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">drop_unspec_vars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 사양서(metacore)에 없는 변수 제거</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">check_variables</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 필수 변수 존재 여부 검증</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">check_ct_data</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, na_acceptable <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># CT(Controlled Terminology) 준수 확인</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">order_cols</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 컬럼 순서 사양서에 맞춤</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sort_by_key</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>       <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 키 변수(SORT KEYS)로 정렬</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, domain <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADAE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 변수 타입 강제 변환 (예: 문자 → 숫자)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_length</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>     <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># SAS 길이 지정 (예: 문자열 200자)</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>      <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 변수 라벨 할당 (예: "Age at Baseline")</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_df_label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 데이터셋 라벨 할당 (예: "Adverse Events Analysis Dataset")</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">xportr_write</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/file.path.html">file.path</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dir</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"adae.xpt"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, metadata <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">metacore</span>, domain <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ADAE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># XPT 파일 저장</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">datatable</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="datatables html-widget html-fill-item" id="htmlwidget-ece3c519e2deb0f66fa9" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-ece3c519e2deb0f66fa9">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144","145","146","147","148","149","150","151","152","153","154","155","156","157","158","159","160","161","162","163","164","165","166","167","168","169","170","171","172","173","174","175","176","177","178","179","180","181","182","183","184","185","186","187","188","189","190","191","192","193","194","195","196","197","198","199","200","201","202","203","204","205","206","207","208","209","210","211","212","213","214","215","216","217","218","219","220","221","222","223","224","225","226","227","228","229","230","231","232","233","234","235","236","237","238","239","240","241","242","243","244","245","246","247","248","249","250","251","252","253","254","255","256","257","258","259","260","261","262","263","264","265","266","267","268","269","270","271","272","273","274","275","276","277","278","279","280","281","282","283","284","285","286","287","288","289","290","291","292","293","294","295","296","297","298","299","300","301","302","303","304","305","306","307","308","309","310","311","312","313","314","315","316","317","318","319","320","321","322","323","324","325","326","327","328","329","330","331","332","333","334","335","336","337","338","339","340","341","342","343","344","345","346","347","348","349","350","351","352","353","354","355","356","357","358","359","360","361","362","363","364","365","366","367","368","369","370","371","372","373","374","375","376","377","378","379","380","381","382","383","384","385","386","387","388","389","390","391","392","393","394","395","396","397","398","399","400","401","402","403","404","405","406","407","408","409","410","411","412","413","414","415","416","417","418","419","420","421","422","423","424","425","426","427","428","429","430","431","432","433","434","435","436","437","438","439","440","441","442","443","444","445","446","447","448","449","450","451","452","453","454","455","456","457","458","459","460","461","462","463","464","465","466","467","468","469","470","471","472","473","474","475","476","477","478","479","480","481","482","483","484","485","486","487","488","489","490","491","492","493","494","495","496","497","498","499","500","501","502","503","504","505","506","507","508","509","510","511","512","513","514","515","516","517","518","519","520","521","522","523","524","525","526","527","528","529","530","531","532","533","534","535","536","537","538","539","540","541","542","543","544","545","546","547","548","549","550","551","552","553","554","555","556","557","558","559","560","561","562","563","564","565","566","567","568","569","570","571","572","573","574","575","576","577","578","579","580","581","582","583","584","585","586","587","588","589","590","591","592","593","594","595","596","597","598","599","600","601","602","603","604","605","606","607","608","609","610","611","612","613","614","615","616","617","618","619","620","621","622","623","624","625","626","627","628","629","630","631","632","633","634","635","636","637","638","639","640","641","642","643","644","645","646","647","648","649","650","651","652","653","654","655","656","657","658","659","660","661","662","663","664","665","666","667","668","669","670","671","672","673","674","675","676","677","678","679","680","681","682","683","684","685","686","687","688","689","690","691","692","693","694","695","696","697","698","699","700","701","702","703","704","705","706","707","708","709","710","711","712","713","714","715","716","717","718","719","720","721","722","723","724","725","726","727","728","729","730","731","732","733","734","735","736","737","738","739","740","741","742","743","744","745","746","747","748","749","750","751","752","753","754","755","756","757","758","759","760","761","762","763","764","765","766","767","768","769","770","771","772","773","774","775","776","777","778","779","780","781","782","783","784","785","786","787","788","789","790","791","792","793","794","795","796","797","798","799","800","801","802","803","804","805","806","807","808","809","810","811","812","813","814","815","816","817","818","819","820","821","822","823","824","825","826","827","828","829","830","831","832","833","834","835","836","837","838","839","840","841","842","843","844","845","846","847","848","849","850","851","852","853","854","855","856","857","858","859","860","861","862","863","864","865","866","867","868","869","870","871","872","873","874","875","876","877","878","879","880","881","882","883","884","885","886","887","888","889","890","891","892","893","894","895","896","897","898","899","900","901","902","903","904","905","906","907","908","909","910","911","912","913","914","915","916","917","918","919","920","921","922","923","924","925","926","927","928","929","930","931","932","933","934","935","936","937","938","939","940","941","942","943","944","945","946","947","948","949","950","951","952","953","954","955","956","957","958","959","960","961","962","963","964","965","966","967","968","969","970","971","972","973","974","975","976","977","978","979","980","981","982","983","984","985","986","987","988","989","990","991","992","993","994","995","996","997","998","999","1000","1001","1002","1003","1004","1005","1006","1007","1008","1009","1010","1011","1012","1013","1014","1015","1016","1017","1018","1019","1020","1021","1022","1023","1024","1025","1026","1027","1028","1029","1030","1031","1032","1033","1034","1035","1036","1037","1038","1039","1040","1041","1042","1043","1044","1045","1046","1047","1048","1049","1050","1051","1052","1053","1054","1055","1056","1057","1058","1059","1060","1061","1062","1063","1064","1065","1066","1067","1068","1069","1070","1071","1072","1073","1074","1075","1076","1077","1078","1079","1080","1081","1082","1083","1084","1085","1086","1087","1088","1089","1090","1091","1092","1093","1094","1095","1096","1097","1098","1099","1100","1101","1102","1103","1104","1105","1106","1107","1108","1109","1110","1111","1112","1113","1114","1115","1116","1117","1118","1119","1120","1121","1122","1123","1124","1125","1126","1127","1128","1129","1130","1131","1132","1133","1134","1135","1136","1137","1138","1139","1140","1141","1142","1143","1144","1145","1146","1147","1148","1149","1150","1151","1152","1153","1154","1155","1156","1157","1158","1159","1160","1161","1162","1163","1164","1165","1166","1167","1168","1169","1170","1171","1172","1173","1174","1175","1176","1177","1178","1179","1180","1181","1182","1183","1184","1185","1186","1187","1188","1189","1190","1191"],["CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01","CDISCPILOT01"],["01-701-1015","01-701-1015","01-701-1015","01-701-1023","01-701-1023","01-701-1023","01-701-1023","01-701-1028","01-701-1028","01-701-1034","01-701-1034","01-701-1047","01-701-1047","01-701-1047","01-701-1047","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1097","01-701-1111","01-701-1111","01-701-1111","01-701-1111","01-701-1111","01-701-1111","01-701-1111","01-701-1111","01-701-1115","01-701-1115","01-701-1115","01-701-1115","01-701-1115","01-701-1115","01-701-1115","01-701-1115","01-701-1115","01-701-1118","01-701-1130","01-701-1130","01-701-1130","01-701-1130","01-701-1130","01-701-1130","01-701-1130","01-701-1130","01-701-1133","01-701-1133","01-701-1133","01-701-1133","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1146","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1148","01-701-1153","01-701-1153","01-701-1180","01-701-1180","01-701-1180","01-701-1180","01-701-1180","01-701-1180","01-701-1180","01-701-1180","01-701-1180","01-701-1181","01-701-1188","01-701-1188","01-701-1188","01-701-1188","01-701-1188","01-701-1188","01-701-1188","01-701-1188","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1192","01-701-1203","01-701-1211","01-701-1211","01-701-1211","01-701-1211","01-701-1211","01-701-1211","01-701-1211","01-701-1211","01-701-1211","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1239","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1275","01-701-1287","01-701-1287","01-701-1287","01-701-1287","01-701-1287","01-701-1294","01-701-1294","01-701-1294","01-701-1294","01-701-1294","01-701-1294","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1302","01-701-1317","01-701-1317","01-701-1317","01-701-1317","01-701-1317","01-701-1317","01-701-1317","01-701-1317","01-701-1317","01-701-1324","01-701-1324","01-701-1324","01-701-1324","01-701-1341","01-701-1341","01-701-1341","01-701-1341","01-701-1341","01-701-1360","01-701-1360","01-701-1360","01-701-1363","01-701-1363","01-701-1363","01-701-1363","01-701-1363","01-701-1363","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1383","01-701-1387","01-701-1387","01-701-1392","01-701-1392","01-701-1415","01-701-1415","01-701-1415","01-701-1415","01-701-1415","01-701-1415","01-701-1442","01-701-1444","01-701-1444","01-701-1444","01-701-1444","01-701-1444","01-701-1444","01-701-1444","01-701-1444","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-702-1082","01-703-1042","01-703-1042","01-703-1076","01-703-1076","01-703-1076","01-703-1076","01-703-1076","01-703-1076","01-703-1076","01-703-1086","01-703-1086","01-703-1086","01-703-1086","01-703-1086","01-703-1086","01-703-1100","01-703-1100","01-703-1100","01-703-1100","01-703-1100","01-703-1100","01-703-1100","01-703-1100","01-703-1100","01-703-1119","01-703-1119","01-703-1119","01-703-1119","01-703-1175","01-703-1182","01-703-1182","01-703-1182","01-703-1182","01-703-1210","01-703-1210","01-703-1210","01-703-1258","01-703-1258","01-703-1258","01-703-1258","01-703-1258","01-703-1258","01-703-1258","01-703-1295","01-703-1299","01-703-1299","01-703-1299","01-703-1299","01-703-1299","01-703-1299","01-703-1299","01-703-1299","01-703-1335","01-703-1335","01-703-1403","01-703-1403","01-703-1403","01-703-1403","01-703-1403","01-703-1439","01-703-1439","01-704-1008","01-704-1008","01-704-1008","01-704-1009","01-704-1009","01-704-1009","01-704-1009","01-704-1009","01-704-1009","01-704-1010","01-704-1010","01-704-1010","01-704-1010","01-704-1010","01-704-1010","01-704-1010","01-704-1017","01-704-1017","01-704-1017","01-704-1017","01-704-1017","01-704-1017","01-704-1017","01-704-1017","01-704-1025","01-704-1025","01-704-1025","01-704-1025","01-704-1025","01-704-1025","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1065","01-704-1074","01-704-1074","01-704-1093","01-704-1093","01-704-1093","01-704-1093","01-704-1114","01-704-1114","01-704-1114","01-704-1114","01-704-1114","01-704-1120","01-704-1120","01-704-1120","01-704-1120","01-704-1120","01-704-1120","01-704-1135","01-704-1135","01-704-1164","01-704-1164","01-704-1164","01-704-1218","01-704-1241","01-704-1241","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1266","01-704-1323","01-704-1323","01-704-1323","01-704-1323","01-704-1325","01-704-1325","01-704-1332","01-704-1332","01-704-1332","01-704-1332","01-704-1332","01-704-1332","01-704-1351","01-704-1351","01-704-1351","01-704-1351","01-704-1388","01-704-1435","01-704-1445","01-705-1031","01-705-1059","01-705-1186","01-705-1199","01-705-1199","01-705-1280","01-705-1281","01-705-1281","01-705-1281","01-705-1292","01-705-1292","01-705-1303","01-705-1303","01-705-1303","01-705-1303","01-705-1310","01-705-1310","01-705-1349","01-705-1349","01-705-1349","01-705-1349","01-705-1393","01-705-1393","01-705-1393","01-705-1393","01-705-1431","01-705-1431","01-706-1041","01-706-1041","01-706-1041","01-706-1041","01-706-1041","01-706-1041","01-706-1041","01-706-1041","01-706-1041","01-706-1049","01-706-1049","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-706-1384","01-707-1206","01-707-1206","01-707-1206","01-707-1206","01-707-1206","01-707-1206","01-707-1206","01-707-1206","01-708-1019","01-708-1019","01-708-1019","01-708-1019","01-708-1019","01-708-1032","01-708-1084","01-708-1084","01-708-1084","01-708-1084","01-708-1084","01-708-1084","01-708-1087","01-708-1087","01-708-1087","01-708-1087","01-708-1087","01-708-1087","01-708-1158","01-708-1158","01-708-1178","01-708-1178","01-708-1178","01-708-1178","01-708-1178","01-708-1178","01-708-1178","01-708-1178","01-708-1178","01-708-1213","01-708-1213","01-708-1216","01-708-1216","01-708-1216","01-708-1216","01-708-1253","01-708-1253","01-708-1253","01-708-1253","01-708-1253","01-708-1253","01-708-1253","01-708-1253","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1272","01-708-1286","01-708-1286","01-708-1286","01-708-1286","01-708-1296","01-708-1296","01-708-1296","01-708-1296","01-708-1296","01-708-1296","01-708-1296","01-708-1297","01-708-1297","01-708-1297","01-708-1316","01-708-1316","01-708-1336","01-708-1336","01-708-1336","01-708-1336","01-708-1336","01-708-1336","01-708-1336","01-708-1347","01-708-1347","01-708-1347","01-708-1347","01-708-1347","01-708-1347","01-708-1347","01-708-1347","01-708-1348","01-708-1348","01-708-1348","01-708-1353","01-708-1353","01-708-1353","01-708-1372","01-708-1406","01-708-1406","01-708-1428","01-708-1428","01-708-1428","01-708-1428","01-708-1428","01-708-1428","01-708-1428","01-709-1007","01-709-1007","01-709-1007","01-709-1007","01-709-1020","01-709-1020","01-709-1020","01-709-1020","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1029","01-709-1081","01-709-1081","01-709-1088","01-709-1088","01-709-1099","01-709-1099","01-709-1099","01-709-1099","01-709-1099","01-709-1099","01-709-1099","01-709-1099","01-709-1102","01-709-1102","01-709-1102","01-709-1102","01-709-1102","01-709-1102","01-709-1168","01-709-1168","01-709-1168","01-709-1168","01-709-1168","01-709-1168","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1217","01-709-1238","01-709-1238","01-709-1238","01-709-1238","01-709-1238","01-709-1238","01-709-1238","01-709-1238","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1259","01-709-1285","01-709-1285","01-709-1285","01-709-1285","01-709-1285","01-709-1285","01-709-1301","01-709-1306","01-709-1306","01-709-1306","01-709-1306","01-709-1306","01-709-1306","01-709-1306","01-709-1306","01-709-1306","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1309","01-709-1312","01-709-1312","01-709-1326","01-709-1326","01-709-1326","01-709-1326","01-709-1326","01-709-1326","01-709-1329","01-709-1329","01-709-1339","01-709-1339","01-709-1424","01-710-1002","01-710-1002","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1006","01-710-1021","01-710-1021","01-710-1021","01-710-1027","01-710-1027","01-710-1027","01-710-1027","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1045","01-710-1053","01-710-1053","01-710-1053","01-710-1053","01-710-1060","01-710-1060","01-710-1060","01-710-1060","01-710-1060","01-710-1060","01-710-1060","01-710-1060","01-710-1070","01-710-1070","01-710-1070","01-710-1070","01-710-1070","01-710-1070","01-710-1077","01-710-1077","01-710-1077","01-710-1077","01-710-1077","01-710-1083","01-710-1137","01-710-1137","01-710-1137","01-710-1137","01-710-1142","01-710-1142","01-710-1142","01-710-1142","01-710-1142","01-710-1154","01-710-1154","01-710-1166","01-710-1166","01-710-1166","01-710-1166","01-710-1166","01-710-1183","01-710-1183","01-710-1187","01-710-1235","01-710-1235","01-710-1235","01-710-1249","01-710-1249","01-710-1249","01-710-1249","01-710-1264","01-710-1264","01-710-1264","01-710-1264","01-710-1264","01-710-1264","01-710-1264","01-710-1264","01-710-1264","01-710-1270","01-710-1270","01-710-1271","01-710-1271","01-710-1271","01-710-1271","01-710-1271","01-710-1278","01-710-1278","01-710-1278","01-710-1278","01-710-1278","01-710-1278","01-710-1278","01-710-1300","01-710-1300","01-710-1300","01-710-1300","01-710-1314","01-710-1314","01-710-1315","01-710-1315","01-710-1315","01-710-1315","01-710-1315","01-710-1315","01-710-1315","01-710-1354","01-710-1354","01-710-1358","01-710-1358","01-710-1358","01-710-1368","01-710-1368","01-710-1368","01-710-1368","01-710-1385","01-710-1385","01-710-1385","01-710-1385","01-710-1385","01-710-1385","01-710-1385","01-710-1385","01-710-1408","01-710-1408","01-710-1408","01-710-1408","01-711-1012","01-711-1012","01-711-1012","01-711-1012","01-711-1012","01-711-1012","01-711-1012","01-711-1036","01-711-1036","01-711-1036","01-711-1036","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1143","01-711-1433","01-711-1433","01-711-1433","01-713-1043","01-713-1043","01-713-1043","01-713-1043","01-713-1043","01-713-1043","01-713-1043","01-713-1073","01-713-1141","01-713-1141","01-713-1141","01-713-1141","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1179","01-713-1209","01-713-1209","01-713-1209","01-713-1209","01-713-1256","01-713-1256","01-713-1269","01-713-1269","01-713-1269","01-713-1269","01-713-1269","01-713-1269","01-713-1448","01-713-1448","01-713-1448","01-713-1448","01-714-1035","01-714-1035","01-714-1035","01-714-1035","01-714-1035","01-714-1035","01-714-1035","01-714-1068","01-714-1068","01-714-1068","01-714-1068","01-714-1068","01-714-1068","01-714-1195","01-714-1195","01-714-1195","01-714-1195","01-714-1195","01-714-1195","01-714-1195","01-714-1195","01-714-1195","01-714-1288","01-714-1288","01-714-1288","01-714-1288","01-714-1288","01-714-1288","01-714-1288","01-714-1288","01-714-1288","01-714-1375","01-714-1375","01-714-1375","01-714-1375","01-714-1375","01-714-1375","01-714-1375","01-714-1375","01-714-1425","01-715-1107","01-715-1107","01-715-1107","01-715-1207","01-715-1319","01-715-1321","01-715-1321","01-715-1321","01-715-1321","01-715-1405","01-715-1405","01-715-1405","01-715-1405","01-715-1405","01-715-1405","01-716-1024","01-716-1024","01-716-1024","01-716-1026","01-716-1026","01-716-1026","01-716-1026","01-716-1026","01-716-1026","01-716-1026","01-716-1044","01-716-1044","01-716-1044","01-716-1063","01-716-1063","01-716-1063","01-716-1063","01-716-1063","01-716-1063","01-716-1063","01-716-1071","01-716-1071","01-716-1071","01-716-1094","01-716-1094","01-716-1094","01-716-1103","01-716-1103","01-716-1103","01-716-1103","01-716-1103","01-716-1108","01-716-1108","01-716-1151","01-716-1151","01-716-1151","01-716-1151","01-716-1157","01-716-1157","01-716-1157","01-716-1157","01-716-1160","01-716-1160","01-716-1167","01-716-1167","01-716-1167","01-716-1167","01-716-1167","01-716-1177","01-716-1189","01-716-1189","01-716-1189","01-716-1189","01-716-1229","01-716-1229","01-716-1298","01-716-1298","01-716-1298","01-716-1298","01-716-1298","01-716-1298","01-716-1308","01-716-1311","01-716-1311","01-716-1311","01-716-1311","01-716-1364","01-716-1364","01-716-1373","01-716-1373","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1418","01-716-1441","01-716-1447","01-716-1447","01-716-1447","01-716-1447","01-716-1447","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1004","01-717-1109","01-717-1109","01-717-1109","01-717-1109","01-717-1109","01-717-1109","01-717-1109","01-717-1109","01-717-1174","01-717-1174","01-717-1174","01-717-1174","01-717-1174","01-717-1174","01-717-1201","01-717-1201","01-717-1344","01-717-1344","01-717-1344","01-717-1344","01-717-1344","01-717-1357","01-717-1357","01-717-1357","01-717-1357","01-717-1357","01-717-1357","01-717-1357","01-717-1357","01-717-1357","01-717-1446","01-717-1446","01-717-1446","01-717-1446","01-717-1446","01-717-1446","01-717-1446","01-717-1446","01-717-1446","01-718-1066","01-718-1066","01-718-1066","01-718-1066","01-718-1079","01-718-1079","01-718-1079","01-718-1101","01-718-1101","01-718-1101","01-718-1101","01-718-1101","01-718-1101","01-718-1139","01-718-1139","01-718-1139","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1150","01-718-1170","01-718-1170","01-718-1170","01-718-1170","01-718-1170","01-718-1170","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1250","01-718-1254","01-718-1254","01-718-1254","01-718-1254","01-718-1254","01-718-1254","01-718-1254","01-718-1254","01-718-1254","01-718-1328","01-718-1328","01-718-1328","01-718-1328","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1355","01-718-1371","01-718-1371","01-718-1371","01-718-1371","01-718-1371","01-718-1371","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427","01-718-1427"],["1015","1015","1015","1023","1023","1023","1023","1028","1028","1034","1034","1047","1047","1047","1047","1097","1097","1097","1097","1097","1097","1097","1097","1097","1097","1111","1111","1111","1111","1111","1111","1111","1111","1115","1115","1115","1115","1115","1115","1115","1115","1115","1118","1130","1130","1130","1130","1130","1130","1130","1130","1133","1133","1133","1133","1146","1146","1146","1146","1146","1146","1146","1146","1146","1146","1146","1148","1148","1148","1148","1148","1148","1148","1148","1148","1148","1153","1153","1180","1180","1180","1180","1180","1180","1180","1180","1180","1181","1188","1188","1188","1188","1188","1188","1188","1188","1192","1192","1192","1192","1192","1192","1192","1192","1192","1192","1192","1192","1192","1192","1192","1203","1211","1211","1211","1211","1211","1211","1211","1211","1211","1239","1239","1239","1239","1239","1239","1239","1239","1239","1239","1275","1275","1275","1275","1275","1275","1275","1275","1275","1275","1275","1275","1275","1275","1275","1287","1287","1287","1287","1287","1294","1294","1294","1294","1294","1294","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1302","1317","1317","1317","1317","1317","1317","1317","1317","1317","1324","1324","1324","1324","1341","1341","1341","1341","1341","1360","1360","1360","1363","1363","1363","1363","1363","1363","1383","1383","1383","1383","1383","1383","1383","1383","1383","1383","1383","1383","1387","1387","1392","1392","1415","1415","1415","1415","1415","1415","1442","1444","1444","1444","1444","1444","1444","1444","1444","1082","1082","1082","1082","1082","1082","1082","1082","1082","1082","1042","1042","1076","1076","1076","1076","1076","1076","1076","1086","1086","1086","1086","1086","1086","1100","1100","1100","1100","1100","1100","1100","1100","1100","1119","1119","1119","1119","1175","1182","1182","1182","1182","1210","1210","1210","1258","1258","1258","1258","1258","1258","1258","1295","1299","1299","1299","1299","1299","1299","1299","1299","1335","1335","1403","1403","1403","1403","1403","1439","1439","1008","1008","1008","1009","1009","1009","1009","1009","1009","1010","1010","1010","1010","1010","1010","1010","1017","1017","1017","1017","1017","1017","1017","1017","1025","1025","1025","1025","1025","1025","1065","1065","1065","1065","1065","1065","1065","1065","1065","1065","1074","1074","1093","1093","1093","1093","1114","1114","1114","1114","1114","1120","1120","1120","1120","1120","1120","1135","1135","1164","1164","1164","1218","1241","1241","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1266","1323","1323","1323","1323","1325","1325","1332","1332","1332","1332","1332","1332","1351","1351","1351","1351","1388","1435","1445","1031","1059","1186","1199","1199","1280","1281","1281","1281","1292","1292","1303","1303","1303","1303","1310","1310","1349","1349","1349","1349","1393","1393","1393","1393","1431","1431","1041","1041","1041","1041","1041","1041","1041","1041","1041","1049","1049","1384","1384","1384","1384","1384","1384","1384","1384","1384","1384","1206","1206","1206","1206","1206","1206","1206","1206","1019","1019","1019","1019","1019","1032","1084","1084","1084","1084","1084","1084","1087","1087","1087","1087","1087","1087","1158","1158","1178","1178","1178","1178","1178","1178","1178","1178","1178","1213","1213","1216","1216","1216","1216","1253","1253","1253","1253","1253","1253","1253","1253","1272","1272","1272","1272","1272","1272","1272","1272","1272","1272","1272","1272","1286","1286","1286","1286","1296","1296","1296","1296","1296","1296","1296","1297","1297","1297","1316","1316","1336","1336","1336","1336","1336","1336","1336","1347","1347","1347","1347","1347","1347","1347","1347","1348","1348","1348","1353","1353","1353","1372","1406","1406","1428","1428","1428","1428","1428","1428","1428","1007","1007","1007","1007","1020","1020","1020","1020","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1029","1081","1081","1088","1088","1099","1099","1099","1099","1099","1099","1099","1099","1102","1102","1102","1102","1102","1102","1168","1168","1168","1168","1168","1168","1217","1217","1217","1217","1217","1217","1217","1217","1217","1217","1217","1238","1238","1238","1238","1238","1238","1238","1238","1259","1259","1259","1259","1259","1259","1259","1259","1259","1259","1259","1285","1285","1285","1285","1285","1285","1301","1306","1306","1306","1306","1306","1306","1306","1306","1306","1309","1309","1309","1309","1309","1309","1309","1309","1309","1309","1309","1309","1309","1309","1309","1312","1312","1326","1326","1326","1326","1326","1326","1329","1329","1339","1339","1424","1002","1002","1006","1006","1006","1006","1006","1006","1006","1006","1006","1006","1006","1006","1021","1021","1021","1027","1027","1027","1027","1045","1045","1045","1045","1045","1045","1045","1045","1045","1045","1045","1045","1045","1053","1053","1053","1053","1060","1060","1060","1060","1060","1060","1060","1060","1070","1070","1070","1070","1070","1070","1077","1077","1077","1077","1077","1083","1137","1137","1137","1137","1142","1142","1142","1142","1142","1154","1154","1166","1166","1166","1166","1166","1183","1183","1187","1235","1235","1235","1249","1249","1249","1249","1264","1264","1264","1264","1264","1264","1264","1264","1264","1270","1270","1271","1271","1271","1271","1271","1278","1278","1278","1278","1278","1278","1278","1300","1300","1300","1300","1314","1314","1315","1315","1315","1315","1315","1315","1315","1354","1354","1358","1358","1358","1368","1368","1368","1368","1385","1385","1385","1385","1385","1385","1385","1385","1408","1408","1408","1408","1012","1012","1012","1012","1012","1012","1012","1036","1036","1036","1036","1143","1143","1143","1143","1143","1143","1143","1143","1143","1143","1143","1143","1143","1143","1433","1433","1433","1043","1043","1043","1043","1043","1043","1043","1073","1141","1141","1141","1141","1179","1179","1179","1179","1179","1179","1179","1179","1179","1179","1179","1179","1179","1179","1179","1209","1209","1209","1209","1256","1256","1269","1269","1269","1269","1269","1269","1448","1448","1448","1448","1035","1035","1035","1035","1035","1035","1035","1068","1068","1068","1068","1068","1068","1195","1195","1195","1195","1195","1195","1195","1195","1195","1288","1288","1288","1288","1288","1288","1288","1288","1288","1375","1375","1375","1375","1375","1375","1375","1375","1425","1107","1107","1107","1207","1319","1321","1321","1321","1321","1405","1405","1405","1405","1405","1405","1024","1024","1024","1026","1026","1026","1026","1026","1026","1026","1044","1044","1044","1063","1063","1063","1063","1063","1063","1063","1071","1071","1071","1094","1094","1094","1103","1103","1103","1103","1103","1108","1108","1151","1151","1151","1151","1157","1157","1157","1157","1160","1160","1167","1167","1167","1167","1167","1177","1189","1189","1189","1189","1229","1229","1298","1298","1298","1298","1298","1298","1308","1311","1311","1311","1311","1364","1364","1373","1373","1418","1418","1418","1418","1418","1418","1418","1418","1418","1418","1441","1447","1447","1447","1447","1447","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1004","1109","1109","1109","1109","1109","1109","1109","1109","1174","1174","1174","1174","1174","1174","1201","1201","1344","1344","1344","1344","1344","1357","1357","1357","1357","1357","1357","1357","1357","1357","1446","1446","1446","1446","1446","1446","1446","1446","1446","1066","1066","1066","1066","1079","1079","1079","1101","1101","1101","1101","1101","1101","1139","1139","1139","1150","1150","1150","1150","1150","1150","1150","1150","1150","1150","1170","1170","1170","1170","1170","1170","1250","1250","1250","1250","1250","1250","1250","1250","1250","1250","1250","1254","1254","1254","1254","1254","1254","1254","1254","1254","1328","1328","1328","1328","1355","1355","1355","1355","1355","1355","1355","1355","1355","1355","1355","1355","1355","1371","1371","1371","1371","1371","1371","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427","1427"],["701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","701","702","702","702","702","702","702","702","702","702","702","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","703","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","704","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","705","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","706","707","707","707","707","707","707","707","707","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","708","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","709","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","710","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","711","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","713","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","714","715","715","715","715","715","715","715","715","715","715","715","715","715","715","715","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","716","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","717","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718","718"],["NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA","NA"],["USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA","USA"],["HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO","NOT HISPANIC OR LATINO"],[63,63,63,64,64,64,64,71,71,77,77,85,85,85,85,68,68,68,68,68,68,68,68,68,68,81,81,81,81,81,81,81,81,84,84,84,84,84,84,84,84,84,52,84,84,84,84,84,84,84,84,81,81,81,81,75,75,75,75,75,75,75,75,75,75,75,57,57,57,57,57,57,57,57,57,57,79,79,56,56,56,56,56,56,56,56,56,79,71,71,71,71,71,71,71,71,80,80,80,80,80,80,80,80,80,80,80,80,80,80,80,81,76,76,76,76,76,76,76,76,76,56,56,56,56,56,56,56,56,56,56,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,56,56,56,56,56,67,67,67,67,67,67,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,61,68,68,68,68,68,68,68,68,68,79,79,79,79,51,51,51,51,51,67,67,67,81,81,81,81,81,81,72,72,72,72,72,72,72,72,72,72,72,72,87,87,78,78,85,85,85,85,85,85,57,63,63,63,63,63,63,63,63,84,84,84,84,84,84,84,84,84,84,64,64,69,69,69,69,69,69,69,71,71,71,71,71,71,84,84,84,84,84,84,84,84,84,81,81,81,81,75,84,84,84,84,72,72,72,78,78,78,78,78,78,78,88,81,81,81,81,81,81,81,81,67,67,67,67,67,67,67,76,76,76,76,76,83,83,83,83,83,83,80,80,80,80,80,80,80,77,77,77,77,77,77,77,77,81,81,81,81,81,81,75,75,75,75,75,75,75,75,75,75,80,80,79,79,79,79,77,77,77,77,77,71,71,71,71,71,71,74,74,67,67,67,81,86,86,82,82,82,82,82,82,82,82,82,82,82,82,82,82,82,82,68,68,68,68,81,81,80,80,80,80,80,80,70,70,70,70,81,74,75,56,66,84,87,87,56,73,73,73,60,60,72,72,72,72,74,74,86,86,86,86,84,84,84,84,68,68,64,64,64,64,64,64,64,64,64,60,60,74,74,74,74,74,74,74,74,74,74,65,65,65,65,65,65,65,65,68,68,68,68,68,62,73,73,73,73,73,73,74,74,74,74,74,74,81,81,77,77,77,77,77,77,77,77,77,76,76,78,78,78,78,61,61,61,61,61,61,61,61,82,82,82,82,82,82,82,82,82,82,82,82,80,80,80,80,57,57,57,57,57,57,57,61,61,61,74,74,73,73,73,73,73,73,73,61,61,61,61,61,61,61,61,79,79,79,87,87,87,84,71,71,84,84,84,84,84,84,84,54,54,54,54,72,72,72,72,82,82,82,82,82,82,82,82,82,82,82,82,82,82,82,82,86,86,69,69,79,79,79,79,79,79,79,79,71,71,71,71,71,71,72,72,72,72,72,72,77,77,77,77,77,77,77,77,77,77,77,69,69,69,69,69,69,69,69,82,82,82,82,82,82,82,82,82,82,82,87,87,87,87,87,87,62,60,60,60,60,60,60,60,60,60,65,65,65,65,65,65,65,65,65,65,65,65,65,65,65,68,68,75,75,75,75,75,75,70,70,81,81,77,88,88,77,77,77,77,77,77,77,77,77,77,77,77,79,79,79,83,83,83,83,83,83,83,83,83,83,83,83,83,83,83,83,83,84,84,84,84,82,82,82,82,82,82,82,82,85,85,85,85,85,85,76,76,76,76,76,89,79,79,79,79,76,76,76,76,76,84,84,81,81,81,81,81,80,80,78,56,56,56,79,79,79,79,78,78,78,78,78,78,78,78,78,83,83,86,86,86,86,86,81,81,81,81,81,81,81,78,78,78,78,78,78,83,83,83,83,83,83,83,73,73,82,82,82,88,88,88,88,77,77,77,77,77,77,77,77,80,80,80,80,67,67,67,67,67,67,67,70,70,70,70,76,76,76,76,76,76,76,76,76,76,76,76,76,76,84,84,84,78,78,78,78,78,78,78,74,79,79,79,79,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,77,77,77,77,71,71,73,73,73,73,73,73,71,71,71,71,88,88,88,88,88,88,88,79,79,79,79,79,79,75,75,75,75,75,75,75,75,75,77,77,77,77,77,77,77,77,77,78,78,78,78,78,78,78,78,81,65,65,65,78,65,75,75,75,75,69,69,69,69,69,69,87,87,87,73,73,73,73,73,73,73,74,74,74,80,80,80,80,80,80,80,78,78,78,82,82,82,79,79,79,79,79,86,86,83,83,83,83,85,85,85,85,83,83,68,68,68,68,68,72,81,81,81,81,73,73,76,76,76,76,76,76,76,78,78,78,78,84,84,74,74,80,80,80,80,80,80,80,80,80,80,85,72,72,72,72,72,80,80,80,80,80,80,80,80,80,80,80,80,80,80,80,80,80,80,80,84,84,84,84,84,84,84,84,73,73,73,73,73,73,85,85,64,64,64,64,64,77,77,77,77,77,77,77,77,77,75,75,75,75,75,75,75,75,75,79,79,79,79,67,67,67,82,82,82,82,82,82,77,77,77,73,73,73,73,73,73,73,73,73,73,80,80,80,80,80,80,82,82,82,82,82,82,82,82,82,82,82,78,78,78,78,78,78,78,78,78,86,86,86,86,79,79,79,79,79,79,79,79,79,79,79,79,79,69,69,69,69,69,69,74,74,74,74,74,74,74,74,74,74,74,74,74,74,74,74],["YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS","YEARS"],["F","F","F","M","M","M","M","M","M","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","F","F","M","M","M","M","M","M","M","M","M","F","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","F","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","F","F","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","M","M","F","F","F","F","F","F","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","M","M","M","M","M","M","M","M","M","M","F","F","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","M","M","M","M","F","M","F","F","F","F","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","F","F","F","M","M","M","M","M","M","M","M","F","F","F","F","F","F","M","M","M","M","F","F","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","M","M","M","M","F","F","M","M","F","F","F","F","F","F","F","F","F","F","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","M","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F","F"],["WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","AMERICAN INDIAN OR ALASKA NATIVE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","WHITE","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN","BLACK OR AFRICAN AMERICAN"],["Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose"],["Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose"],["2014-01-02","2014-01-02","2014-01-02","2012-08-05","2012-08-05","2012-08-05","2012-08-05","2013-07-19","2013-07-19","2014-07-01","2014-07-01","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2014-03-12","2014-02-15","2014-02-15","2014-02-15","2014-02-15","2014-02-15","2014-02-15","2014-02-15","2014-02-15","2012-10-28","2012-10-28","2012-10-28","2012-10-28","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-05-20","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-08-23","2013-09-23","2013-09-23","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-12-05","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2012-07-22","2013-02-02","2012-11-15","2012-11-15","2012-11-15","2012-11-15","2012-11-15","2012-11-15","2012-11-15","2012-11-15","2012-11-15","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-02-07","2014-01-25","2014-01-25","2014-01-25","2014-01-25","2014-01-25","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2014-05-22","2014-05-22","2014-05-22","2014-05-22","2014-05-22","2014-05-22","2014-05-22","2014-05-22","2014-05-22","2012-10-02","2012-10-02","2012-10-02","2012-10-02","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-07-31","2013-07-31","2013-07-31","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2013-02-04","2014-03-12","2014-03-12","2012-10-28","2012-10-28","2013-09-23","2013-09-23","2013-09-23","2013-09-23","2013-09-23","2013-09-23","2013-10-26","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-07-26","2013-03-02","2013-03-02","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2012-09-02","2012-09-02","2012-09-02","2012-09-02","2012-09-02","2012-09-02","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-02-20","2013-02-20","2013-02-20","2013-02-20","2013-12-20","2013-10-17","2013-10-17","2013-10-17","2013-10-17","2013-03-16","2013-03-16","2013-03-16","2012-07-20","2012-07-20","2012-07-20","2012-07-20","2012-07-20","2012-07-20","2012-07-20","2013-11-21","2012-09-12","2012-09-12","2012-09-12","2012-09-12","2012-09-12","2012-09-12","2012-09-12","2012-09-12","2014-03-17","2014-03-17","2012-12-12","2012-12-12","2012-12-12","2012-12-12","2012-12-12","2014-03-12","2014-03-12","2013-01-13","2013-01-13","2013-01-13","2013-08-27","2013-08-27","2013-08-27","2013-08-27","2013-08-27","2013-08-27","2014-02-21","2014-02-21","2014-02-21","2014-02-21","2014-02-21","2014-02-21","2014-02-21","2013-10-06","2013-10-06","2013-10-06","2013-10-06","2013-10-06","2013-10-06","2013-10-06","2013-10-06","2013-09-27","2013-09-27","2013-09-27","2013-09-27","2013-09-27","2013-09-27","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2014-01-22","2014-01-22","2013-03-15","2013-03-15","2013-03-15","2013-03-15","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-12-02","2013-12-02","2013-12-02","2013-12-02","2013-12-02","2013-12-02","2013-10-31","2013-10-31","2012-09-19","2012-09-19","2012-09-19","2012-11-19","2013-08-25","2013-08-25","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-07-08","2013-07-08","2013-07-08","2013-07-08","2014-04-23","2014-04-23","2013-12-09","2013-12-09","2013-12-09","2013-12-09","2013-12-09","2013-12-09","2013-10-12","2013-10-12","2013-10-12","2013-10-12","2012-12-07","2012-11-17","2014-05-11","2013-11-27","2013-08-05","2014-01-08","2013-09-16","2013-09-16","2014-01-17","2013-11-28","2013-11-28","2013-11-28","2013-10-14","2013-10-14","2013-12-16","2013-12-16","2013-12-16","2013-12-16","2013-11-02","2013-11-02","2013-03-10","2013-03-10","2013-03-10","2013-03-10","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2013-06-23","2013-06-23","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-05-14","2013-05-14","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2013-10-28","2013-10-28","2013-10-28","2013-10-28","2013-10-28","2013-10-28","2013-10-28","2013-10-28","2013-12-20","2013-12-20","2013-12-20","2013-12-20","2013-12-20","2013-02-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2012-10-22","2012-10-22","2012-10-22","2012-10-22","2012-10-22","2012-10-22","2014-02-08","2014-02-08","2013-12-29","2013-12-29","2013-12-29","2013-12-29","2013-12-29","2013-12-29","2013-12-29","2013-12-29","2013-12-29","2013-02-09","2013-02-09","2012-10-24","2012-10-24","2012-10-24","2012-10-24","2013-05-07","2013-05-07","2013-05-07","2013-05-07","2013-05-07","2013-05-07","2013-05-07","2013-05-07","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-02-06","2013-09-10","2013-09-10","2013-09-10","2013-09-10","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-01-25","2013-01-25","2013-01-25","2013-08-23","2013-08-23","2012-12-07","2012-12-07","2012-12-07","2012-12-07","2012-12-07","2012-12-07","2012-12-07","2013-04-20","2013-04-20","2013-04-20","2013-04-20","2013-04-20","2013-04-20","2013-04-20","2013-04-20","2013-08-05","2013-08-05","2013-08-05","2013-07-04","2013-07-04","2013-07-04","2013-04-12","2013-12-26","2013-12-26","2013-11-09","2013-11-09","2013-11-09","2013-11-09","2013-11-09","2013-11-09","2013-11-09","2012-07-31","2012-07-31","2012-07-31","2012-07-31","2012-12-01","2012-12-01","2012-12-01","2012-12-01","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2012-12-25","2014-01-18","2014-01-18","2014-04-12","2014-04-12","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-10-25","2013-01-15","2013-01-15","2013-01-15","2013-01-15","2013-01-15","2013-01-15","2013-08-02","2013-08-02","2013-08-02","2013-08-02","2013-08-02","2013-08-02","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-03-04","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-07-19","2014-02-03","2014-02-03","2014-02-03","2014-02-03","2014-02-03","2014-02-03","2014-02-03","2014-02-03","2014-02-03","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2014-04-20","2014-04-20","2013-04-05","2013-04-05","2013-04-05","2013-04-05","2013-04-05","2013-04-05","2013-08-15","2013-08-15","2012-12-23","2012-12-23","2013-03-03","2014-01-14","2014-01-14","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-09-27","2013-09-27","2013-09-27","2014-02-28","2014-02-28","2014-02-28","2014-02-28","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2012-12-26","2012-12-26","2012-12-26","2012-12-26","2013-01-01","2013-01-01","2013-01-01","2013-01-01","2013-01-01","2013-01-01","2013-01-01","2013-01-01","2012-09-08","2012-09-08","2012-09-08","2012-09-08","2012-09-08","2012-09-08","2013-11-17","2013-11-17","2013-11-17","2013-11-17","2013-11-17","2013-07-22","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2012-10-02","2012-10-02","2012-10-02","2012-10-02","2012-10-02","2014-03-29","2014-03-29","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2012-11-30","2013-11-16","2013-11-16","2012-11-10","2012-09-26","2012-09-26","2012-09-26","2013-12-28","2013-12-28","2013-12-28","2013-12-28","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2014-02-12","2014-02-12","2012-09-27","2012-09-27","2012-09-27","2012-09-27","2012-09-27","2012-12-24","2012-12-24","2012-12-24","2012-12-24","2012-12-24","2012-12-24","2012-12-24","2012-12-15","2012-12-15","2012-12-15","2012-12-15","2013-04-24","2013-04-24","2013-02-27","2013-02-27","2013-02-27","2013-02-27","2013-02-27","2013-02-27","2013-02-27","2012-11-11","2012-11-11","2012-09-17","2012-09-17","2012-09-17","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2012-10-29","2012-10-29","2012-10-29","2012-10-29","2012-10-29","2012-10-29","2012-10-29","2012-10-29","2013-01-05","2013-01-05","2013-01-05","2013-01-05","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2012-07-29","2012-07-29","2012-07-29","2012-07-29","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-04-03","2013-01-27","2013-01-27","2013-01-27","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2014-03-30","2013-05-31","2013-05-31","2013-05-31","2013-05-31","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-05-27","2013-05-27","2013-05-27","2013-05-27","2012-09-19","2012-09-19","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-19","2014-01-19","2014-01-19","2014-01-19","2014-04-17","2014-04-17","2014-04-17","2014-04-17","2014-04-17","2014-04-17","2014-04-17","2013-08-08","2013-08-08","2013-08-08","2013-08-08","2013-08-08","2013-08-08","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-12-04","2013-12-04","2013-12-04","2013-12-04","2013-12-04","2013-12-04","2013-12-04","2013-12-04","2013-12-04","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-08-15","2013-02-26","2013-02-26","2013-02-26","2012-11-18","2013-02-17","2014-02-11","2014-02-11","2014-02-11","2014-02-11","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2012-07-09","2012-07-09","2012-07-09","2014-04-02","2014-04-02","2014-04-02","2014-04-02","2014-04-02","2014-04-02","2014-04-02","2013-04-27","2013-04-27","2013-04-27","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-06-08","2013-06-08","2013-06-08","2012-12-19","2012-12-19","2012-12-19","2014-03-20","2014-03-20","2014-03-20","2014-03-20","2014-03-20","2013-02-12","2013-02-12","2013-02-01","2013-02-01","2013-02-01","2013-02-01","2013-10-02","2013-10-02","2013-10-02","2013-10-02","2013-04-05","2013-04-05","2012-10-08","2012-10-08","2012-10-08","2012-10-08","2012-10-08","2014-09-02","2012-10-09","2012-10-09","2012-10-09","2012-10-09","2013-02-20","2013-02-20","2013-04-08","2013-04-08","2013-04-08","2013-04-08","2013-04-08","2013-04-08","2013-08-28","2014-05-14","2014-05-14","2014-05-14","2014-05-14","2013-07-04","2013-07-04","2012-12-14","2012-12-14","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2013-05-05","2014-01-22","2013-12-16","2013-12-16","2013-12-16","2013-12-16","2013-12-16","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2014-01-27","2013-01-22","2013-01-22","2013-01-22","2013-01-22","2013-01-22","2013-01-22","2013-12-19","2013-12-19","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2013-05-01","2013-05-01","2013-05-01","2013-05-01","2013-05-01","2013-05-01","2013-05-01","2013-05-01","2013-05-01","2013-09-01","2013-09-01","2013-09-01","2013-09-01","2013-09-01","2013-09-01","2013-09-01","2013-09-01","2013-09-01","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2012-09-19","2012-09-19","2012-09-19","2013-02-17","2013-02-17","2013-02-17","2013-02-17","2013-02-17","2013-02-17","2013-05-19","2013-05-19","2013-05-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-09-21","2013-07-10","2013-07-10","2013-07-10","2013-07-10","2013-07-10","2013-07-10","2013-07-10","2013-07-10","2013-07-10","2013-02-01","2013-02-01","2013-02-01","2013-02-01","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-02-28","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2013-04-26","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17","2012-12-17"],["2014-01-02T00:00:00Z","2014-01-02T00:00:00Z","2014-01-02T00:00:00Z","2012-08-05T00:00:00Z","2012-08-05T00:00:00Z","2012-08-05T00:00:00Z","2012-08-05T00:00:00Z","2013-07-19T00:00:00Z","2013-07-19T00:00:00Z","2014-07-01T00:00:00Z","2014-07-01T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2014-01-01T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2014-03-12T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2014-02-15T00:00:00Z","2012-10-28T00:00:00Z","2012-10-28T00:00:00Z","2012-10-28T00:00:00Z","2012-10-28T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-05-20T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2013-09-23T00:00:00Z","2013-09-23T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-12-05T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2013-02-15T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2012-07-22T00:00:00Z","2013-02-02T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2012-11-15T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-02-07T00:00:00Z","2014-01-25T00:00:00Z","2014-01-25T00:00:00Z","2014-01-25T00:00:00Z","2014-01-25T00:00:00Z","2014-01-25T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2013-08-29T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2014-05-22T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-07-31T00:00:00Z","2013-07-31T00:00:00Z","2013-07-31T00:00:00Z","2013-05-30T00:00:00Z","2013-05-30T00:00:00Z","2013-05-30T00:00:00Z","2013-05-30T00:00:00Z","2013-05-30T00:00:00Z","2013-05-30T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2013-02-04T00:00:00Z","2014-03-12T00:00:00Z","2014-03-12T00:00:00Z","2012-10-28T00:00:00Z","2012-10-28T00:00:00Z","2013-09-23T00:00:00Z","2013-09-23T00:00:00Z","2013-09-23T00:00:00Z","2013-09-23T00:00:00Z","2013-09-23T00:00:00Z","2013-09-23T00:00:00Z","2013-10-26T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-07-26T00:00:00Z","2013-03-02T00:00:00Z","2013-03-02T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2012-09-02T00:00:00Z","2012-09-02T00:00:00Z","2012-09-02T00:00:00Z","2012-09-02T00:00:00Z","2012-09-02T00:00:00Z","2012-09-02T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-03-13T00:00:00Z","2013-02-20T00:00:00Z","2013-02-20T00:00:00Z","2013-02-20T00:00:00Z","2013-02-20T00:00:00Z","2013-12-20T00:00:00Z","2013-10-17T00:00:00Z","2013-10-17T00:00:00Z","2013-10-17T00:00:00Z","2013-10-17T00:00:00Z","2013-03-16T00:00:00Z","2013-03-16T00:00:00Z","2013-03-16T00:00:00Z","2012-07-20T00:00:00Z","2012-07-20T00:00:00Z","2012-07-20T00:00:00Z","2012-07-20T00:00:00Z","2012-07-20T00:00:00Z","2012-07-20T00:00:00Z","2012-07-20T00:00:00Z","2013-11-21T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2012-09-12T00:00:00Z","2014-03-17T00:00:00Z","2014-03-17T00:00:00Z","2012-12-12T00:00:00Z","2012-12-12T00:00:00Z","2012-12-12T00:00:00Z","2012-12-12T00:00:00Z","2012-12-12T00:00:00Z","2014-03-12T00:00:00Z","2014-03-12T00:00:00Z","2013-01-13T00:00:00Z","2013-01-13T00:00:00Z","2013-01-13T00:00:00Z","2013-08-27T00:00:00Z","2013-08-27T00:00:00Z","2013-08-27T00:00:00Z","2013-08-27T00:00:00Z","2013-08-27T00:00:00Z","2013-08-27T00:00:00Z","2014-02-21T00:00:00Z","2014-02-21T00:00:00Z","2014-02-21T00:00:00Z","2014-02-21T00:00:00Z","2014-02-21T00:00:00Z","2014-02-21T00:00:00Z","2014-02-21T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-10-06T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2013-10-24T00:00:00Z","2014-01-22T00:00:00Z","2014-01-22T00:00:00Z","2013-03-15T00:00:00Z","2013-03-15T00:00:00Z","2013-03-15T00:00:00Z","2013-03-15T00:00:00Z","2013-01-23T00:00:00Z","2013-01-23T00:00:00Z","2013-01-23T00:00:00Z","2013-01-23T00:00:00Z","2013-01-23T00:00:00Z","2013-12-02T00:00:00Z","2013-12-02T00:00:00Z","2013-12-02T00:00:00Z","2013-12-02T00:00:00Z","2013-12-02T00:00:00Z","2013-12-02T00:00:00Z","2013-10-31T00:00:00Z","2013-10-31T00:00:00Z","2012-09-19T00:00:00Z","2012-09-19T00:00:00Z","2012-09-19T00:00:00Z","2012-11-19T00:00:00Z","2013-08-25T00:00:00Z","2013-08-25T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-10-13T00:00:00Z","2013-07-08T00:00:00Z","2013-07-08T00:00:00Z","2013-07-08T00:00:00Z","2013-07-08T00:00:00Z","2014-04-23T00:00:00Z","2014-04-23T00:00:00Z","2013-12-09T00:00:00Z","2013-12-09T00:00:00Z","2013-12-09T00:00:00Z","2013-12-09T00:00:00Z","2013-12-09T00:00:00Z","2013-12-09T00:00:00Z","2013-10-12T00:00:00Z","2013-10-12T00:00:00Z","2013-10-12T00:00:00Z","2013-10-12T00:00:00Z","2012-12-07T00:00:00Z","2012-11-17T00:00:00Z","2014-05-11T00:00:00Z","2013-11-27T00:00:00Z","2013-08-05T00:00:00Z","2014-01-08T00:00:00Z","2013-09-16T00:00:00Z","2013-09-16T00:00:00Z","2014-01-17T00:00:00Z","2013-11-28T00:00:00Z","2013-11-28T00:00:00Z","2013-11-28T00:00:00Z","2013-10-14T00:00:00Z","2013-10-14T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2013-11-02T00:00:00Z","2013-11-02T00:00:00Z","2013-03-10T00:00:00Z","2013-03-10T00:00:00Z","2013-03-10T00:00:00Z","2013-03-10T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2012-09-07T00:00:00Z","2013-06-23T00:00:00Z","2013-06-23T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-12-31T00:00:00Z","2013-05-14T00:00:00Z","2013-05-14T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2012-09-15T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-10-28T00:00:00Z","2013-12-20T00:00:00Z","2013-12-20T00:00:00Z","2013-12-20T00:00:00Z","2013-12-20T00:00:00Z","2013-12-20T00:00:00Z","2013-02-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2012-10-22T00:00:00Z","2012-10-22T00:00:00Z","2012-10-22T00:00:00Z","2012-10-22T00:00:00Z","2012-10-22T00:00:00Z","2012-10-22T00:00:00Z","2014-02-08T00:00:00Z","2014-02-08T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-12-29T00:00:00Z","2013-02-09T00:00:00Z","2013-02-09T00:00:00Z","2012-10-24T00:00:00Z","2012-10-24T00:00:00Z","2012-10-24T00:00:00Z","2012-10-24T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-05-07T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-02-06T00:00:00Z","2013-09-10T00:00:00Z","2013-09-10T00:00:00Z","2013-09-10T00:00:00Z","2013-09-10T00:00:00Z","2013-06-14T00:00:00Z","2013-06-14T00:00:00Z","2013-06-14T00:00:00Z","2013-06-14T00:00:00Z","2013-06-14T00:00:00Z","2013-06-14T00:00:00Z","2013-06-14T00:00:00Z","2013-01-25T00:00:00Z","2013-01-25T00:00:00Z","2013-01-25T00:00:00Z","2013-08-23T00:00:00Z","2013-08-23T00:00:00Z","2012-12-07T00:00:00Z","2012-12-07T00:00:00Z","2012-12-07T00:00:00Z","2012-12-07T00:00:00Z","2012-12-07T00:00:00Z","2012-12-07T00:00:00Z","2012-12-07T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-04-20T00:00:00Z","2013-08-05T00:00:00Z","2013-08-05T00:00:00Z","2013-08-05T00:00:00Z","2013-07-04T00:00:00Z","2013-07-04T00:00:00Z","2013-07-04T00:00:00Z","2013-04-12T00:00:00Z","2013-12-26T00:00:00Z","2013-12-26T00:00:00Z","2013-11-09T00:00:00Z","2013-11-09T00:00:00Z","2013-11-09T00:00:00Z","2013-11-09T00:00:00Z","2013-11-09T00:00:00Z","2013-11-09T00:00:00Z","2013-11-09T00:00:00Z","2012-07-31T00:00:00Z","2012-07-31T00:00:00Z","2012-07-31T00:00:00Z","2012-07-31T00:00:00Z","2012-12-01T00:00:00Z","2012-12-01T00:00:00Z","2012-12-01T00:00:00Z","2012-12-01T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2012-12-25T00:00:00Z","2014-01-18T00:00:00Z","2014-01-18T00:00:00Z","2014-04-12T00:00:00Z","2014-04-12T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-10-25T00:00:00Z","2013-01-15T00:00:00Z","2013-01-15T00:00:00Z","2013-01-15T00:00:00Z","2013-01-15T00:00:00Z","2013-01-15T00:00:00Z","2013-01-15T00:00:00Z","2013-08-02T00:00:00Z","2013-08-02T00:00:00Z","2013-08-02T00:00:00Z","2013-08-02T00:00:00Z","2013-08-02T00:00:00Z","2013-08-02T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-03-04T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-05-15T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-01-26T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-03-24T00:00:00Z","2013-07-19T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2014-02-03T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2013-06-19T00:00:00Z","2014-04-20T00:00:00Z","2014-04-20T00:00:00Z","2013-04-05T00:00:00Z","2013-04-05T00:00:00Z","2013-04-05T00:00:00Z","2013-04-05T00:00:00Z","2013-04-05T00:00:00Z","2013-04-05T00:00:00Z","2013-08-15T00:00:00Z","2013-08-15T00:00:00Z","2012-12-23T00:00:00Z","2012-12-23T00:00:00Z","2013-03-03T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-02-10T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2013-09-27T00:00:00Z","2014-02-28T00:00:00Z","2014-02-28T00:00:00Z","2014-02-28T00:00:00Z","2014-02-28T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2013-06-03T00:00:00Z","2012-12-26T00:00:00Z","2012-12-26T00:00:00Z","2012-12-26T00:00:00Z","2012-12-26T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2013-01-01T00:00:00Z","2012-09-08T00:00:00Z","2012-09-08T00:00:00Z","2012-09-08T00:00:00Z","2012-09-08T00:00:00Z","2012-09-08T00:00:00Z","2012-09-08T00:00:00Z","2013-11-17T00:00:00Z","2013-11-17T00:00:00Z","2013-11-17T00:00:00Z","2013-11-17T00:00:00Z","2013-11-17T00:00:00Z","2013-07-22T00:00:00Z","2013-10-11T00:00:00Z","2013-10-11T00:00:00Z","2013-10-11T00:00:00Z","2013-10-11T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2012-10-02T00:00:00Z","2014-03-29T00:00:00Z","2014-03-29T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2012-11-30T00:00:00Z","2013-11-16T00:00:00Z","2013-11-16T00:00:00Z","2012-11-10T00:00:00Z","2012-09-26T00:00:00Z","2012-09-26T00:00:00Z","2012-09-26T00:00:00Z","2013-12-28T00:00:00Z","2013-12-28T00:00:00Z","2013-12-28T00:00:00Z","2013-12-28T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2013-06-13T00:00:00Z","2014-02-12T00:00:00Z","2014-02-12T00:00:00Z","2012-09-27T00:00:00Z","2012-09-27T00:00:00Z","2012-09-27T00:00:00Z","2012-09-27T00:00:00Z","2012-09-27T00:00:00Z","2012-12-24T00:00:00Z","2012-12-24T00:00:00Z","2012-12-24T00:00:00Z","2012-12-24T00:00:00Z","2012-12-24T00:00:00Z","2012-12-24T00:00:00Z","2012-12-24T00:00:00Z","2012-12-15T00:00:00Z","2012-12-15T00:00:00Z","2012-12-15T00:00:00Z","2012-12-15T00:00:00Z","2013-04-24T00:00:00Z","2013-04-24T00:00:00Z","2013-02-27T00:00:00Z","2013-02-27T00:00:00Z","2013-02-27T00:00:00Z","2013-02-27T00:00:00Z","2013-02-27T00:00:00Z","2013-02-27T00:00:00Z","2013-02-27T00:00:00Z","2012-11-11T00:00:00Z","2012-11-11T00:00:00Z","2012-09-17T00:00:00Z","2012-09-17T00:00:00Z","2012-09-17T00:00:00Z","2013-10-23T00:00:00Z","2013-10-23T00:00:00Z","2013-10-23T00:00:00Z","2013-10-23T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2012-10-29T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-01-05T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2012-07-29T00:00:00Z","2012-07-29T00:00:00Z","2012-07-29T00:00:00Z","2012-07-29T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-04-03T00:00:00Z","2013-01-27T00:00:00Z","2013-01-27T00:00:00Z","2013-01-27T00:00:00Z","2013-09-15T00:00:00Z","2013-09-15T00:00:00Z","2013-09-15T00:00:00Z","2013-09-15T00:00:00Z","2013-09-15T00:00:00Z","2013-09-15T00:00:00Z","2013-09-15T00:00:00Z","2014-03-30T00:00:00Z","2013-05-31T00:00:00Z","2013-05-31T00:00:00Z","2013-05-31T00:00:00Z","2013-05-31T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-08-09T00:00:00Z","2013-05-27T00:00:00Z","2013-05-27T00:00:00Z","2013-05-27T00:00:00Z","2013-05-27T00:00:00Z","2012-09-19T00:00:00Z","2012-09-19T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-19T00:00:00Z","2014-01-19T00:00:00Z","2014-01-19T00:00:00Z","2014-01-19T00:00:00Z","2014-04-17T00:00:00Z","2014-04-17T00:00:00Z","2014-04-17T00:00:00Z","2014-04-17T00:00:00Z","2014-04-17T00:00:00Z","2014-04-17T00:00:00Z","2014-04-17T00:00:00Z","2013-08-08T00:00:00Z","2013-08-08T00:00:00Z","2013-08-08T00:00:00Z","2013-08-08T00:00:00Z","2013-08-08T00:00:00Z","2013-08-08T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-12-04T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-08-15T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2013-02-26T00:00:00Z","2012-11-18T00:00:00Z","2013-02-17T00:00:00Z","2014-02-11T00:00:00Z","2014-02-11T00:00:00Z","2014-02-11T00:00:00Z","2014-02-11T00:00:00Z","2013-07-06T00:00:00Z","2013-07-06T00:00:00Z","2013-07-06T00:00:00Z","2013-07-06T00:00:00Z","2013-07-06T00:00:00Z","2013-07-06T00:00:00Z","2012-07-09T00:00:00Z","2012-07-09T00:00:00Z","2012-07-09T00:00:00Z","2014-04-02T00:00:00Z","2014-04-02T00:00:00Z","2014-04-02T00:00:00Z","2014-04-02T00:00:00Z","2014-04-02T00:00:00Z","2014-04-02T00:00:00Z","2014-04-02T00:00:00Z","2013-04-27T00:00:00Z","2013-04-27T00:00:00Z","2013-04-27T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-05-09T00:00:00Z","2013-06-08T00:00:00Z","2013-06-08T00:00:00Z","2013-06-08T00:00:00Z","2012-12-19T00:00:00Z","2012-12-19T00:00:00Z","2012-12-19T00:00:00Z","2014-03-20T00:00:00Z","2014-03-20T00:00:00Z","2014-03-20T00:00:00Z","2014-03-20T00:00:00Z","2014-03-20T00:00:00Z","2013-02-12T00:00:00Z","2013-02-12T00:00:00Z","2013-02-01T00:00:00Z","2013-02-01T00:00:00Z","2013-02-01T00:00:00Z","2013-02-01T00:00:00Z","2013-10-02T00:00:00Z","2013-10-02T00:00:00Z","2013-10-02T00:00:00Z","2013-10-02T00:00:00Z","2013-04-05T00:00:00Z","2013-04-05T00:00:00Z","2012-10-08T00:00:00Z","2012-10-08T00:00:00Z","2012-10-08T00:00:00Z","2012-10-08T00:00:00Z","2012-10-08T00:00:00Z","2014-09-02T00:00:00Z","2012-10-09T00:00:00Z","2012-10-09T00:00:00Z","2012-10-09T00:00:00Z","2012-10-09T00:00:00Z","2013-02-20T00:00:00Z","2013-02-20T00:00:00Z","2013-04-08T00:00:00Z","2013-04-08T00:00:00Z","2013-04-08T00:00:00Z","2013-04-08T00:00:00Z","2013-04-08T00:00:00Z","2013-04-08T00:00:00Z","2013-08-28T00:00:00Z","2014-05-14T00:00:00Z","2014-05-14T00:00:00Z","2014-05-14T00:00:00Z","2014-05-14T00:00:00Z","2013-07-04T00:00:00Z","2013-07-04T00:00:00Z","2012-12-14T00:00:00Z","2012-12-14T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2013-05-05T00:00:00Z","2014-01-22T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2013-12-16T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-14T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2014-01-27T00:00:00Z","2013-01-22T00:00:00Z","2013-01-22T00:00:00Z","2013-01-22T00:00:00Z","2013-01-22T00:00:00Z","2013-01-22T00:00:00Z","2013-01-22T00:00:00Z","2013-12-19T00:00:00Z","2013-12-19T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2014-01-11T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-05-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-09-01T00:00:00Z","2013-07-07T00:00:00Z","2013-07-07T00:00:00Z","2013-07-07T00:00:00Z","2013-07-07T00:00:00Z","2012-09-19T00:00:00Z","2012-09-19T00:00:00Z","2012-09-19T00:00:00Z","2013-02-17T00:00:00Z","2013-02-17T00:00:00Z","2013-02-17T00:00:00Z","2013-02-17T00:00:00Z","2013-02-17T00:00:00Z","2013-02-17T00:00:00Z","2013-05-19T00:00:00Z","2013-05-19T00:00:00Z","2013-05-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-01-19T00:00:00Z","2013-09-16T00:00:00Z","2013-09-16T00:00:00Z","2013-09-16T00:00:00Z","2013-09-16T00:00:00Z","2013-09-16T00:00:00Z","2013-09-16T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-09-21T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-07-10T00:00:00Z","2013-02-01T00:00:00Z","2013-02-01T00:00:00Z","2013-02-01T00:00:00Z","2013-02-01T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-02-28T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2013-04-26T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z","2012-12-17T00:00:00Z"],["2014-07-02","2014-07-02","2014-07-02","2012-09-01","2012-09-01","2012-09-01","2012-09-01","2014-01-14","2014-01-14","2014-12-30","2014-12-30","2013-03-09","2013-03-09","2013-03-09","2013-03-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2014-07-09","2012-09-16","2012-09-16","2012-09-16","2012-09-16","2012-09-16","2012-09-16","2012-09-16","2012-09-16","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2013-01-23","2014-09-09","2014-08-16","2014-08-16","2014-08-16","2014-08-16","2014-08-16","2014-08-16","2014-08-16","2014-08-16","2013-04-28","2013-04-28","2013-04-28","2013-04-28","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-02-20","2014-03-16","2014-03-16","2013-03-18","2013-03-18","2013-03-18","2013-03-18","2013-03-18","2013-03-18","2013-03-18","2013-03-18","2013-03-18","2013-12-09","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-03-24","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-01-20","2013-08-03","2013-01-12","2013-01-12","2013-01-12","2013-01-12","2013-01-12","2013-01-12","2013-01-12","2013-01-12","2013-01-12","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-07-10","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-05-31","2014-07-26","2014-07-26","2014-07-26","2014-07-26","2014-07-26","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-06-14","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2014-11-20","2014-11-20","2014-11-20","2014-11-20","2014-11-20","2014-11-20","2014-11-20","2014-11-20","2014-11-20","2013-04-02","2013-04-02","2013-04-02","2013-04-02","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-01-26","2013-08-05","2013-08-05","2013-08-05","2013-11-27","2013-11-27","2013-11-27","2013-11-27","2013-11-27","2013-11-27","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2014-03-25","2014-03-25","2013-04-28","2013-04-28","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-04-26","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-02-12","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-10-13","2013-08-31","2013-08-31","2013-12-24","2013-12-24","2013-12-24","2013-12-24","2013-12-24","2013-12-24","2013-12-24","2012-12-04","2012-12-04","2012-12-04","2012-12-04","2012-12-04","2012-12-04","2013-09-14","2013-09-14","2013-09-14","2013-09-14","2013-09-14","2013-09-14","2013-09-14","2013-09-14","2013-09-14","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-12-26","2013-12-11","2013-12-11","2013-12-11","2013-12-11","2013-09-06","2013-09-06","2013-09-06","2013-01-11","2013-01-11","2013-01-11","2013-01-11","2013-01-11","2013-01-11","2013-01-11","2014-04-19","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2014-05-07","2014-05-07","2012-12-13","2012-12-13","2012-12-13","2012-12-13","2012-12-13","2014-09-11","2014-09-11","2013-02-21","2013-02-21","2013-02-21","2013-09-25","2013-09-25","2013-09-25","2013-09-25","2013-09-25","2013-09-25","2014-07-08","2014-07-08","2014-07-08","2014-07-08","2014-07-08","2014-07-08","2014-07-08","2013-11-18","2013-11-18","2013-11-18","2013-11-18","2013-11-18","2013-11-18","2013-11-18","2013-11-18","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-10-24","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2013-12-22","2014-03-20","2014-03-20","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2014-02-01","2014-02-01","2014-02-01","2014-02-01","2014-02-01","2014-02-01","2014-05-16","2014-05-16","2013-04-04","2013-04-04","2013-04-04","2013-05-27","2013-10-09","2013-10-09","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-12-06","2013-08-05","2013-08-05","2013-08-05","2013-08-05","2014-07-04","2014-07-04","2014-02-14","2014-02-14","2014-02-14","2014-02-14","2014-02-14","2014-02-14","2014-04-18","2014-04-18","2014-04-18","2014-04-18","2013-06-11","2013-01-09","2014-11-01","2013-12-18","2013-12-05","2014-01-26","2013-09-28","2013-09-28","2014-07-25","2014-02-27","2014-02-27","2014-02-27","2014-05-13","2014-05-13","2013-12-30","2013-12-30","2013-12-30","2013-12-30","2014-01-23","2014-01-23","2013-09-08","2013-09-08","2013-09-08","2013-09-08","2013-02-01","2013-02-01","2013-02-01","2013-02-01","2013-12-19","2013-12-19","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2013-06-18","2013-06-18","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2012-09-24","2014-04-26","2014-04-26","2014-04-26","2014-04-26","2014-04-26","2014-04-26","2014-04-26","2014-04-26","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2013-03-01","2013-11-11","2013-11-11","2013-11-11","2013-11-11","2013-11-11","2013-11-11","2013-04-28","2013-04-28","2013-04-28","2013-04-28","2013-04-28","2013-04-28","2014-03-21","2014-03-21","2014-04-06","2014-04-06","2014-04-06","2014-04-06","2014-04-06","2014-04-06","2014-04-06","2014-04-06","2014-04-06","2013-02-22","2013-02-22","2012-11-29","2012-11-29","2012-11-29","2012-11-29","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2013-03-22","2014-03-08","2014-03-08","2014-03-08","2014-03-08","2013-12-12","2013-12-12","2013-12-12","2013-12-12","2013-12-12","2013-12-12","2013-12-12","2013-05-03","2013-05-03","2013-05-03","2014-03-01","2014-03-01","2013-06-05","2013-06-05","2013-06-05","2013-06-05","2013-06-05","2013-06-05","2013-06-05","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2014-02-13","2014-02-13","2014-02-13","2013-08-28","2013-08-28","2013-08-28","2013-04-19","2014-07-02","2014-07-02","2013-12-14","2013-12-14","2013-12-14","2013-12-14","2013-12-14","2013-12-14","2013-12-14","2012-08-28","2012-08-28","2012-08-28","2012-08-28","2013-06-01","2013-06-01","2013-06-01","2013-06-01","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2013-06-26","2014-04-27","2014-04-27","2014-10-09","2014-10-09","2014-04-25","2014-04-25","2014-04-25","2014-04-25","2014-04-25","2014-04-25","2014-04-25","2014-04-25","2013-03-27","2013-03-27","2013-03-27","2013-03-27","2013-03-27","2013-03-27","2013-09-26","2013-09-26","2013-09-26","2013-09-26","2013-09-26","2013-09-26","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-06-11","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-08-06","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2014-01-17","2014-06-16","2014-06-16","2014-06-16","2014-06-16","2014-06-16","2014-06-16","2014-06-16","2014-06-16","2014-06-16","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2013-12-19","2014-10-19","2014-10-19","2013-10-02","2013-10-02","2013-10-02","2013-10-02","2013-10-02","2013-10-02","2013-08-25","2013-08-25","2013-06-24","2013-06-24","2013-03-07","2014-01-18","2014-01-18","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-08-09","2013-10-29","2013-10-29","2013-10-29","2014-08-29","2014-08-29","2014-08-29","2014-08-29","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-08-13","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-07-05","2013-07-05","2013-07-05","2013-07-05","2013-07-05","2013-07-05","2013-07-05","2013-07-05","2013-01-22","2013-01-22","2013-01-22","2013-01-22","2013-01-22","2013-01-22","2014-05-18","2014-05-18","2014-05-18","2014-05-18","2014-05-18","2013-08-01","2013-11-13","2013-11-13","2013-11-13","2013-11-13","2012-10-20","2012-10-20","2012-10-20","2012-10-20","2012-10-20","2014-04-27","2014-04-27","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2014-05-17","2014-05-17","2013-05-12","2013-03-27","2013-03-27","2013-03-27","2014-06-28","2014-06-28","2014-06-28","2014-06-28","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2014-03-01","2014-03-01","2012-11-21","2012-11-21","2012-11-21","2012-11-21","2012-11-21","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-26","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2013-05-23","2013-05-23","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-05-01","2013-05-01","2013-02-09","2013-02-09","2013-02-09","2014-04-24","2014-04-24","2014-04-24","2014-04-24","2013-02-18","2013-02-18","2013-02-18","2013-02-18","2013-02-18","2013-02-18","2013-02-18","2013-02-18","2013-07-12","2013-07-12","2013-07-12","2013-07-12","2013-04-29","2013-04-29","2013-04-29","2013-04-29","2013-04-29","2013-04-29","2013-04-29","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-05-30","2013-02-05","2013-02-05","2013-02-05","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2014-10-05","2013-07-01","2013-07-01","2013-07-01","2013-07-01","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2013-11-22","2013-11-22","2013-11-22","2013-11-22","2013-03-25","2013-03-25","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-05-16","2014-05-16","2014-05-16","2014-05-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2013-10-08","2013-10-08","2013-10-08","2013-10-08","2013-10-08","2013-10-08","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2013-10-23","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2013-09-07","2013-09-07","2013-09-07","2013-09-07","2013-09-07","2013-09-07","2013-09-07","2013-09-07","2013-08-19","2013-05-07","2013-05-07","2013-05-07","2013-05-27","2013-03-05","2014-04-21","2014-04-21","2014-04-21","2014-04-21","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-01-20","2013-01-20","2013-01-20","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2014-10-16","2013-11-03","2013-11-03","2013-11-03","2013-08-25","2013-08-25","2013-08-25","2013-08-25","2013-08-25","2013-08-25","2013-08-25","2013-08-01","2013-08-01","2013-08-01","2013-01-24","2013-01-24","2013-01-24","2014-09-20","2014-09-20","2014-09-20","2014-09-20","2014-09-20","2013-08-10","2013-08-10","2013-05-11","2013-05-11","2013-05-11","2013-05-11","2014-04-04","2014-04-04","2014-04-04","2014-04-04","2013-10-11","2013-10-11","2013-04-12","2013-04-12","2013-04-12","2013-04-12","2013-04-12","2015-03-05","2013-02-27","2013-02-27","2013-02-27","2013-02-27","2013-03-31","2013-03-31","2013-06-28","2013-06-28","2013-06-28","2013-06-28","2013-06-28","2013-06-28","2013-10-07","2014-09-21","2014-09-21","2014-09-21","2014-09-21","2014-01-09","2014-01-09","2013-02-27","2013-02-27","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2013-11-20","2014-07-22","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-06-17","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-16","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2014-07-28","2013-07-24","2013-07-24","2013-07-24","2013-07-24","2013-07-24","2013-07-24","2014-02-21","2014-02-21","2014-03-14","2014-03-14","2014-03-14","2014-03-14","2014-03-14","2013-10-14","2013-10-14","2013-10-14","2013-10-14","2013-10-14","2013-10-14","2013-10-14","2013-10-14","2013-10-14","2014-03-03","2014-03-03","2014-03-03","2014-03-03","2014-03-03","2014-03-03","2014-03-03","2014-03-03","2014-03-03","2013-07-16","2013-07-16","2013-07-16","2013-07-16","2012-10-31","2012-10-31","2012-10-31","2013-07-31","2013-07-31","2013-07-31","2013-07-31","2013-07-31","2013-07-31","2013-11-17","2013-11-17","2013-11-17","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-07-29","2013-10-12","2013-10-12","2013-10-12","2013-10-12","2013-10-12","2013-10-12","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-31","2014-01-09","2014-01-09","2014-01-09","2014-01-09","2014-01-09","2014-01-09","2014-01-09","2014-01-09","2014-01-09","2013-04-18","2013-04-18","2013-04-18","2013-04-18","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-29","2013-08-01","2013-08-01","2013-08-01","2013-08-01","2013-08-01","2013-08-01","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-02-11"],["Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y"],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-01-14","2013-01-14","2013-01-14","2013-01-14","2013-01-14","2013-01-14","2013-01-14","2013-01-14","2013-01-14",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2014-11-01",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-08-02",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],["Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline Low Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Placebo","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose","Xanomeline High Dose"],[1,2,3,1,2,4,3,1,2,1,2,1,2,3,4,1,3,2,4,10,5,6,9,8,7,3,1,4,2,5,6,7,8,1,5,2,6,3,4,7,9,8,1,2,1,3,5,4,6,8,7,1,3,2,4,1,2,3,4,5,7,6,8,9,11,10,8,9,1,2,3,4,5,6,7,10,1,2,4,1,2,3,6,5,9,8,7,1,1,3,2,4,5,6,7,8,4,9,1,3,6,2,5,8,11,7,10,13,12,15,14,1,1,2,3,4,5,6,7,8,9,1,6,2,3,4,5,7,8,9,10,1,11,2,3,12,5,14,6,15,4,13,7,9,8,10,1,2,3,4,5,1,3,5,2,4,6,1,2,3,4,5,9,8,18,10,6,7,11,12,15,13,14,16,17,19,20,21,23,22,2,1,3,4,5,6,7,8,9,1,2,3,4,2,1,4,5,3,1,2,3,2,4,1,3,5,6,2,1,3,4,7,5,6,8,9,10,11,12,1,2,1,2,1,2,3,4,5,6,1,1,4,2,5,3,6,7,8,2,6,3,4,1,5,7,8,9,10,1,2,3,1,2,5,4,7,6,1,2,3,6,4,5,6,8,1,2,3,4,5,7,9,1,2,3,4,1,1,2,3,4,1,2,3,2,5,1,3,4,6,7,1,3,1,2,4,6,5,7,8,1,2,1,5,2,3,4,1,2,1,2,3,1,2,4,3,5,6,1,3,2,6,7,4,5,1,2,4,3,6,8,5,7,2,1,3,5,4,6,1,3,2,4,6,5,7,8,9,10,1,2,1,2,4,3,1,2,4,3,5,1,3,2,6,4,5,1,2,1,2,3,1,1,2,1,2,9,12,3,5,4,13,6,10,7,14,8,15,11,16,1,4,3,2,1,2,1,3,5,2,6,4,2,4,1,3,1,1,1,1,1,1,1,2,1,2,1,3,1,2,1,3,2,4,1,2,1,2,4,3,2,4,1,3,1,2,1,7,2,3,5,4,8,6,9,1,2,1,6,3,8,2,7,4,9,5,10,1,6,2,4,3,5,7,8,1,5,2,3,4,1,1,2,3,4,5,6,1,2,3,4,5,6,1,2,1,2,3,5,4,9,7,8,6,1,2,2,4,1,3,2,5,1,4,3,6,7,8,1,2,3,12,4,5,8,6,9,7,10,11,1,2,3,4,1,2,3,4,5,6,7,1,2,3,1,2,1,2,3,5,7,4,6,3,7,4,8,2,6,1,5,1,2,3,1,2,3,1,1,2,1,3,6,2,5,4,7,1,2,3,4,1,3,2,4,1,4,2,3,5,6,8,7,9,10,13,11,14,12,15,16,1,2,1,2,1,5,2,6,4,7,3,8,1,3,2,4,5,6,1,2,3,5,4,6,1,2,3,8,10,4,5,6,7,9,11,1,2,4,3,5,6,7,8,1,2,3,5,6,4,8,9,7,10,11,1,2,5,3,6,4,1,2,4,8,1,3,7,5,9,6,1,10,2,13,5,6,3,4,7,8,9,11,12,14,15,1,2,1,2,4,3,5,6,1,2,1,2,1,1,2,1,2,3,4,6,5,7,8,9,12,11,10,1,2,3,1,3,2,4,1,2,5,4,10,3,9,6,11,7,12,8,13,1,3,2,4,1,2,3,4,5,6,7,8,1,3,2,4,5,6,4,5,1,2,3,1,2,1,3,4,1,2,3,4,5,1,2,4,1,2,3,5,2,1,1,1,2,3,1,2,3,4,1,2,3,5,4,7,6,9,8,1,2,3,1,2,4,5,1,2,6,3,7,4,5,1,3,2,4,1,2,1,2,5,4,7,3,6,1,2,2,1,3,1,4,2,3,1,2,3,4,6,5,7,8,1,2,3,4,1,3,2,4,7,5,6,1,2,4,3,1,2,3,5,6,4,7,8,9,11,10,14,12,13,1,2,3,1,4,2,5,3,6,7,1,1,2,3,4,1,2,3,5,4,6,7,9,10,8,15,12,14,11,13,1,3,2,4,1,2,1,2,3,5,4,6,1,3,2,4,1,5,2,4,3,6,7,1,2,3,5,4,6,1,6,2,7,3,8,4,5,9,1,2,3,5,4,6,8,7,9,1,3,2,4,5,7,6,8,1,1,3,2,1,1,1,2,3,4,4,3,2,1,5,6,1,2,3,1,2,3,6,4,5,7,1,2,3,1,2,3,4,5,7,6,2,1,3,1,2,3,2,3,1,4,5,1,2,1,4,3,2,1,4,2,3,2,1,1,2,3,4,5,1,2,1,3,4,1,2,1,4,2,5,3,6,1,1,4,2,3,1,2,1,2,2,1,3,4,5,7,6,8,9,10,1,1,2,3,4,5,1,2,10,7,3,12,5,8,4,9,6,11,13,14,15,16,17,18,19,1,8,3,2,4,5,6,7,1,5,2,6,3,4,1,2,2,3,1,4,5,1,2,5,3,4,9,6,7,8,1,2,3,4,9,5,6,7,8,1,2,4,3,2,3,1,1,2,4,3,6,5,1,2,3,1,4,2,5,3,6,9,7,8,10,1,3,2,4,5,6,1,4,6,2,3,5,7,8,9,10,11,1,2,3,4,5,6,7,9,8,1,2,3,4,3,1,2,4,5,6,7,8,9,10,11,12,13,1,2,3,6,5,4,1,2,3,7,4,5,6,8,10,9,11,12,13,15,14,16],["APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","DIARRHOEA","ERYTHEMA","ERYTHEMA","ERYTHEMA","ATRIOVENTRICULAR BLOCK SECOND DEGREE","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","FATIGUE","HIATUS HERNIA","HIATUS HERNIA","UPPER RESPIRATORY TRACT INFECTION","BUNDLE BRANCH BLOCK LEFT","ERYTHEMA","APPLICATION SITE VESICLES","PRURITUS GENERALISED","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","PRURITUS GENERALISED","PRURITUS GENERALISED","NASAL CONGESTION","PHARYNGOLARYNGEAL PAIN","PRURITUS GENERALISED","LOCALISED INFECTION","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","MICTURITION URGENCY","ARTHRALGIA","CELLULITIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","FATIGUE","APPLICATION SITE DERMATITIS","APPLICATION SITE IRRITATION","COUGH","PYREXIA","URINARY TRACT INFECTION","URINARY TRACT INFECTION","EYE ALLERGY","EYE SWELLING","EYE PRURITUS","NASAL CONGESTION","PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","RASH","RASH","APPLICATION SITE IRRITATION","FATIGUE","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PAIN","APPLICATION SITE IRRITATION","APPLICATION SITE VESICLES","DYSPEPSIA","DEPRESSED MOOD","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","LOWER RESPIRATORY TRACT INFECTION","LOWER RESPIRATORY TRACT INFECTION","FLANK PAIN","CALCULUS URETHRAL","EPISTAXIS","ACTINIC KERATOSIS","INCREASED APPETITE","INCREASED APPETITE","HEADACHE","VOMITING","VOMITING","MICTURITION URGENCY","MICTURITION URGENCY","NASOPHARYNGITIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","AGITATION","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","URTICARIA","URTICARIA","APPLICATION SITE URTICARIA","URTICARIA","COUGH","COUGH","NASAL MUCOSA BIOPSY","ERYTHEMA","ERYTHEMA","SECRETION DISCHARGE","SECRETION DISCHARGE","MUSCULAR WEAKNESS","MUSCULAR WEAKNESS","PNEUMONIA","PNEUMONIA","APPLICATION SITE ERYTHEMA","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE PRURITUS","EYE LASER SURGERY","APPLICATION SITE PRURITUS","JOINT DISLOCATION","SHOULDER PAIN","INCONTINENCE","APPLICATION SITE ERYTHEMA","SKIN LACERATION","CONFUSIONAL STATE","DYSPNOEA","SUDDEN DEATH","ERYTHEMA","ERYTHEMA","SKIN ODOUR ABNORMAL","PRURITUS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","FATIGUE","HORDEOLUM","NAUSEA","NAUSEA","VOMITING","ERYTHEMA","ERYTHEMA","PAROSMIA","PAROSMIA","PRURITUS","PRURITUS","SALIVARY HYPERSECRETION","SALIVARY HYPERSECRETION","COUGH","COUGH","NASAL CONGESTION","NASAL CONGESTION","SALIVARY HYPERSECRETION","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","RHINORRHOEA","RHINORRHOEA","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","RASH ERYTHEMATOUS","APPLICATION SITE PERSPIRATION","HYPERHIDROSIS","STOMACH DISCOMFORT","PAIN","NASAL CONGESTION","DIZZINESS","MALAISE","MALAISE","MYALGIA","PHARYNGOLARYNGEAL PAIN","PRODUCTIVE COUGH","DIZZINESS","CONTUSION","CONTUSION","HEADACHE","EPISTAXIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","RESPIRATORY TRACT CONGESTION","RHINITIS","FEELING ABNORMAL","LIBIDO DECREASED","LISTLESS","CYST","ONYCHOMYCOSIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE URTICARIA","DEPRESSED MOOD","DEPRESSED MOOD","FATIGUE","DYSPHONIA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","APPLICATION SITE VESICLES","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE IRRITATION","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","HEADACHE","HEADACHE","NAUSEA","APPLICATION SITE PRURITUS","BACK PAIN","BACK PAIN","APPLICATION SITE PAIN","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE VESICLES","CHEST DISCOMFORT","HEADACHE","COUGH","DIARRHOEA","HYPERHIDROSIS","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","MICTURITION URGENCY","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","DIARRHOEA","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","SALIVARY HYPERSECRETION","APPLICATION SITE IRRITATION","APPLICATION SITE VESICLES","PARAESTHESIA","NEUTROPHIL COUNT INCREASED","NEUTROPHIL COUNT INCREASED","URINE ANALYSIS ABNORMAL","URINE ANALYSIS ABNORMAL","WHITE BLOOD CELL COUNT INCREASED","WHITE BLOOD CELL COUNT INCREASED","RECTAL HAEMORRHAGE","RECTAL HAEMORRHAGE","APPLICATION SITE IRRITATION","SKIN IRRITATION","DIARRHOEA","INSOMNIA","HYPERCHOLESTEROLAEMIA","BIOPSY PROSTATE","BENIGN PROSTATIC HYPERPLASIA","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","HYPERHIDROSIS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","CHILLS","CHILLS","OEDEMA PERIPHERAL","OEDEMA PERIPHERAL","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","CATARACT OPERATION","UPPER RESPIRATORY TRACT INFECTION","OEDEMA PERIPHERAL","OEDEMA PERIPHERAL","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","PRURITUS","PRURITUS","ATRIOVENTRICULAR BLOCK SECOND DEGREE","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","SOMNOLENCE","SOMNOLENCE","DISORIENTATION","DECREASED APPETITE","DECREASED APPETITE","DIZZINESS","DIZZINESS","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","ARTHRITIS","GLAUCOMA","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","HYPOTENSION","HYPOTENSION","TACHYCARDIA","TACHYCARDIA","MYOCARDIAL INFARCTION","ATRIOVENTRICULAR BLOCK SECOND DEGREE","ATRIOVENTRICULAR BLOCK SECOND DEGREE","WEIGHT DECREASED","WEIGHT DECREASED","DIARRHOEA","VOMITING","CHEST DISCOMFORT","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","HYPERHIDROSIS","TRANSIENT ISCHAEMIC ATTACK","SINUS BRADYCARDIA","FATIGUE","MALAISE","MALAISE","RASH","DIZZINESS","NAUSEA","CYSTITIS","DIARRHOEA","VOMITING","ARTHRALGIA","CONTUSION","EXCORIATION","SKIN LACERATION","MYOCARDIAL INFARCTION","VENTRICULAR SEPTAL DEFECT","CARDIAC DISORDER","INFLUENZA","PRURITUS","PRURITUS","RASH","RASH","PALPITATIONS","VERTIGO","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","WOLFF-PARKINSON-WHITE SYNDROME","WOLFF-PARKINSON-WHITE SYNDROME","ABDOMINAL DISCOMFORT","APPLICATION SITE PRURITUS","RASH","APPLICATION SITE IRRITATION","HYPERHIDROSIS","MALAISE","CHILLS","DIZZINESS","DIARRHOEA","SINUS BRADYCARDIA","PRURITUS","ERYTHEMA","BLOOD GLUCOSE INCREASED","BLOOD GLUCOSE INCREASED","PRURITUS","RASH","ERYTHEMA","OEDEMA","PRURITUS","RASH","INFLUENZA","RASH","RASH","NASOPHARYNGITIS","NASOPHARYNGITIS","PRURITUS","PRURITUS","RASH","RASH","INFLUENZA","INFLUENZA","OEDEMA PERIPHERAL","BLOOD GLUCOSE INCREASED","RASH","DYSPNOEA","SEASONAL ALLERGY","ERYTHEMA","ERYTHEMA","ERYTHEMA","ATRIAL FLUTTER","ATRIAL FLUTTER","ATRIAL FIBRILLATION","ATRIAL FIBRILLATION","HYPERHIDROSIS","HYPERHIDROSIS","BLISTER","BLISTER","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","VIRAL INFECTION","OEDEMA","PRURITUS","RASH","RASH","PRURITUS","RASH","RASH","RASH","PRURITUS","PRURITUS","BURNING SENSATION","FLATULENCE","FLATULENCE","POLLAKIURIA","POLLAKIURIA","HEADACHE","PARKINSON'S DISEASE","COMPLETED SUICIDE","RASH","ERYTHEMA","HYPERBILIRUBINAEMIA","DYSPEPSIA","DYSPEPSIA","PRURITUS","EXCORIATION","PRURITUS","PRURITUS","PRURITUS","ERYTHEMA","PRURITUS","PRURITUS","RASH","RASH","RASH PRURITIC","WOUND HAEMORRHAGE","NAUSEA","VOMITING","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","VENTRICULAR HYPERTROPHY","PRURITUS","PRURITUS","ABDOMINAL PAIN","COLON CANCER","RASH ERYTHEMATOUS","ANXIETY","ANXIETY","ANXIETY","DIARRHOEA","EXCORIATION","EXCORIATION","IRRITABILITY","IRRITABILITY","PRURITUS","PRURITUS","SYNCOPE","SYNCOPE","ERYTHEMA","ERYTHEMA","EXCORIATION","EXCORIATION","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","SINUS ARRHYTHMIA","SINUS ARRHYTHMIA","BLOOD CREATINE PHOSPHOKINASE INCREASED","BLOOD CREATINE PHOSPHOKINASE INCREASED","POSTNASAL DRIP","POSTNASAL DRIP","CONJUNCTIVITIS","CONJUNCTIVITIS","HEADACHE","DIZZINESS","STUPOR","TRANSIENT ISCHAEMIC ATTACK","VISION BLURRED","APPLICATION SITE VESICLES","DIZZINESS","DIZZINESS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","SINUS BRADYCARDIA","SINUS BRADYCARDIA","HEART RATE IRREGULAR","HEART RATE IRREGULAR","DIZZINESS","HEADACHE","HEART RATE IRREGULAR","HEART RATE IRREGULAR","AGITATION","ERYTHEMA","FLANK PAIN","NASOPHARYNGITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","VENTRICULAR SEPTAL DEFECT","ALCOHOL USE","ATRIAL FIBRILLATION","CYSTITIS","DIZZINESS","ATRIAL FIBRILLATION","ATRIAL FIBRILLATION","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE SWELLING","CONJUNCTIVITIS","CONJUNCTIVITIS","COUGH","COUGH","HEART RATE INCREASED","HEART RATE INCREASED","DIZZINESS","DIZZINESS","DIARRHOEA","MUSCLE SPASMS","VENTRICULAR EXTRASYSTOLES","VENTRICULAR EXTRASYSTOLES","APPLICATION SITE DERMATITIS","COUGH","COUGH","NASOPHARYNGITIS","NASOPHARYNGITIS","HEADACHE","HEADACHE","HOT FLUSH","APPLICATION SITE PRURITUS","MYOCARDIAL INFARCTION","ABDOMINAL PAIN","CONSTIPATION","ATRIAL HYPERTROPHY","ATRIAL HYPERTROPHY","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","APPLICATION SITE VESICLES","PAIN IN EXTREMITY","APPLICATION SITE PRURITUS","APPLICATION SITE SWELLING","SINUS BRADYCARDIA","HEADACHE","APPLICATION SITE ERYTHEMA","DIZZINESS","APPLICATION SITE PRURITUS","DIZZINESS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE SWELLING","APPLICATION SITE SWELLING","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","APPLICATION SITE PRURITUS","DIZZINESS","DIZZINESS","HYPOTENSION","ATRIOVENTRICULAR BLOCK SECOND DEGREE","OEDEMA PERIPHERAL","OEDEMA PERIPHERAL","EPISTAXIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE WARMTH","APPLICATION SITE WARMTH","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","DIARRHOEA","APPLICATION SITE DERMATITIS","ABDOMINAL PAIN","DIARRHOEA","APPLICATION SITE PRURITUS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","APPLICATION SITE ERYTHEMA","MUSCLE SPASMS","MUSCLE SPASMS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","SINUS BRADYCARDIA","SINUS BRADYCARDIA","COUGH","COUGH","PHARYNGEAL ERYTHEMA","PHARYNGEAL ERYTHEMA","MYOCARDIAL INFARCTION","SINUS BRADYCARDIA","PRURITUS","RASH","OEDEMA PERIPHERAL","APPLICATION SITE DERMATITIS","RASH","NASOPHARYNGITIS","NASOPHARYNGITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","ACROCHORDON EXCISION","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","DIZZINESS","APPLICATION SITE VESICLES","SKIN IRRITATION","SKIN IRRITATION","PRURITUS","PRURITUS","RASH","RASH","DIZZINESS","COUGH","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","DIZZINESS","ELECTROCARDIOGRAM T WAVE INVERSION","DIZZINESS","DIZZINESS","HEADACHE","PRURITUS","DIARRHOEA","APPLICATION SITE DERMATITIS","APPLICATION SITE DISCHARGE","APPLICATION SITE PRURITUS","HIP FRACTURE","COUGH","UPPER RESPIRATORY TRACT INFECTION","SKIN LESION EXCISION","LOCALISED INFECTION","LOCALISED INFECTION","BLOOD URINE PRESENT","BENIGN PROSTATIC HYPERPLASIA","BENIGN PROSTATIC HYPERPLASIA","CYSTOSCOPY","DEHYDRATION","HYPOTENSION","PYREXIA","ORTHOSTATIC HYPOTENSION","ORTHOSTATIC HYPOTENSION","HEMIANOPIA HOMONYMOUS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","SINUS BRADYCARDIA","DIABETES MELLITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","RASH","RASH","RASH","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","DRUG ERUPTION","RASH MACULO-PAPULAR","RASH MACULO-PAPULAR","RASH PAPULAR","RASH PAPULAR","NASAL CONGESTION","NIGHTMARE","VOMITING","VOMITING","RASH MACULO-PAPULAR","BIOPSY","DIZZINESS","DIZZINESS","ALLERGIC GRANULOMATOUS ANGIITIS","APPLICATION SITE PRURITUS","FEELING COLD","APPLICATION SITE DERMATITIS","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","SYNCOPE","VOMITING","APPLICATION SITE BLEEDING","COUGH","DIZZINESS","PYREXIA","HEADACHE","BLOOD ALKALINE PHOSPHATASE INCREASED","SYNCOPE","TRANSIENT ISCHAEMIC ATTACK","TRANSIENT ISCHAEMIC ATTACK","DIZZINESS","SALIVARY HYPERSECRETION","SKIN IRRITATION","DIZZINESS","NEPHROLITHIASIS","URINARY TRACT INFECTION","PRURITUS","SINUS BRADYCARDIA","SINUS BRADYCARDIA","CONTUSION","FACIAL BONES FRACTURE","FALL","CHEST PAIN","PRURITUS","PRURITUS","HAEMOPTYSIS","HAEMOPTYSIS","RALES","RALES","VOMITING","NASOPHARYNGITIS","NASOPHARYNGITIS","PRURITUS","PRURITUS","SKIN IRRITATION","SKIN IRRITATION","RASH","RASH","ERYTHEMA","ERYTHEMA","SKIN EXFOLIATION","SKIN EXFOLIATION","PRURITUS","PRURITUS","RASH","RASH","NASAL CONGESTION","INSOMNIA","INSOMNIA","APPLICATION SITE INDURATION","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","FALL","FALL","SUPRAVENTRICULAR EXTRASYSTOLES","ERYTHEMA","PRURITUS","RASH","HIP FRACTURE","GASTROINTESTINAL HAEMORRHAGE","ARTHRITIS","ARTHRITIS","APPLICATION SITE DERMATITIS","SKIN IRRITATION","AGITATION","MYOCARDIAL INFARCTION","ERYTHEMA","PRURITUS","VERTIGO","HEADACHE","DIZZINESS","HEADACHE","NAUSEA","MYOCARDIAL INFARCTION","SYNCOPE VASOVAGAL","SKIN IRRITATION","SKIN IRRITATION","COMPLEX PARTIAL SEIZURES","FALL","SKIN LACERATION","SYNCOPE","SYNCOPE","ATRIOVENTRICULAR BLOCK FIRST DEGREE","SINUS BRADYCARDIA","APPLICATION SITE IRRITATION","COUGH","SHOULDER PAIN","APPLICATION SITE PRURITUS","PRURITUS","PRURITUS","ERYTHEMA","PRURITUS","DIARRHOEA","PRURITUS","RASH","CHILLS","COLD SWEAT","CHILLS","COLD SWEAT","CHILLS","COLD SWEAT","ERYTHEMA","PRURITUS","ATRIAL FIBRILLATION","DYSPNOEA","MYOCARDIAL INFARCTION","CARDIAC FAILURE CONGESTIVE","HYPONATRAEMIA","SOMNOLENCE","RASH","RASH","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","BURNING SENSATION","VOMITING","PRURITUS","BLISTER","ERYTHEMA","APPLICATION SITE IRRITATION","NASOPHARYNGITIS","NASOPHARYNGITIS","HEADACHE","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","RASH","RASH","ERYTHEMA","PRURITUS","PRURITUS","RASH","APPLICATION SITE DESQUAMATION","HIP FRACTURE","HIP FRACTURE","DELUSION","PSYCHOMOTOR HYPERACTIVITY","SUPRAVENTRICULAR TACHYCARDIA","SUPRAVENTRICULAR TACHYCARDIA","ATRIOVENTRICULAR BLOCK FIRST DEGREE","PRURITUS","BLISTER","RASH","ATRIAL FIBRILLATION","ELECTROCARDIOGRAM T WAVE AMPLITUDE DECREASED","NASOPHARYNGITIS","NASOPHARYNGITIS","ERYTHEMA","ERYTHEMA","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","NASOPHARYNGITIS","APPLICATION SITE URTICARIA","HEADACHE","EAR INFECTION","EAR INFECTION","GLOSSITIS","TINNITUS","PARAESTHESIA ORAL","NASOPHARYNGITIS","NASOPHARYNGITIS","NASOPHARYNGITIS","SUPRAVENTRICULAR EXTRASYSTOLES","SUPRAVENTRICULAR EXTRASYSTOLES","APPLICATION SITE DERMATITIS","RASH PRURITIC","RASH PRURITIC","VENTRICULAR EXTRASYSTOLES","VENTRICULAR EXTRASYSTOLES","CONJUNCTIVAL HAEMORRHAGE","CERUMEN IMPACTION","HYPERTENSION","HYPERTENSION","ASTHENIA","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","NAUSEA","NAUSEA","VOMITING","VOMITING","CATARACT OPERATION","APPLICATION SITE PRURITUS","RASH PRURITIC","RASH PRURITIC","BACK PAIN","DELIRIUM","DYSURIA","DYSPEPSIA","URINARY TRACT INFECTION","URINARY TRACT INFECTION","ASTHENIA","ASTHENIA","DYSPEPSIA","GASTROENTERITIS VIRAL","GASTROOESOPHAGEAL REFLUX DISEASE","PELVIC PAIN","PELVIC PAIN","CERVICITIS","CERVICITIS","VAGINAL MYCOSIS","VAGINAL MYCOSIS","ABDOMINAL PAIN","ABDOMINAL PAIN","APPLICATION SITE PRURITUS","DIZZINESS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE ERYTHEMA","COUGH","SHOULDER PAIN","SHOULDER PAIN","RASH","RASH","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","NAUSEA","NAUSEA","BUNDLE BRANCH BLOCK RIGHT","BUNDLE BRANCH BLOCK RIGHT","ELECTROCARDIOGRAM T WAVE INVERSION","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","SUPRAVENTRICULAR EXTRASYSTOLES","SUPRAVENTRICULAR EXTRASYSTOLES","BRONCHITIS","SKIN IRRITATION","SKIN IRRITATION","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","DIARRHOEA","DIARRHOEA","HYPERHIDROSIS","HYPERHIDROSIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","SOMNOLENCE","APPLICATION SITE PRURITUS","DYSURIA","DIZZINESS","DIARRHOEA","ERYTHEMA","COUGH","NASOPHARYNGITIS","BACK PAIN","BACK PAIN","ARTHRALGIA","SINUS BRADYCARDIA","SKIN IRRITATION","SKIN IRRITATION","EAR PAIN","EAR PAIN","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","ATRIOVENTRICULAR BLOCK SECOND DEGREE","DERMATITIS CONTACT","DERMATITIS CONTACT","SINUS BRADYCARDIA","NEPHROLITHIASIS","SINUS BRADYCARDIA","NAUSEA","NAUSEA","APPLICATION SITE REACTION","SINUS BRADYCARDIA","CONFUSIONAL STATE","DIZZINESS","HYPERHIDROSIS","PALPITATIONS","SINUS BRADYCARDIA","SINUS BRADYCARDIA","NASOPHARYNGITIS","NASOPHARYNGITIS","EMPHYSEMA","HYPERHIDROSIS","PRURITUS","ELECTROCARDIOGRAM T WAVE INVERSION","ELECTROCARDIOGRAM T WAVE INVERSION","EAR INFECTION","ELECTROCARDIOGRAM T WAVE AMPLITUDE DECREASED","EAR INFECTION","DIARRHOEA","DIARRHOEA","CONFUSIONAL STATE","HYPERHIDROSIS","FATIGUE","PRURITUS","PRURITUS","RESTLESSNESS","RESTLESSNESS","LETHARGY","FATIGUE","HYPERSOMNIA","SYNCOPE","ATRIAL FLUTTER","BUNDLE BRANCH BLOCK RIGHT","VENTRICULAR SEPTAL DEFECT","BLISTER","ERYTHEMA","PRURITUS","BLISTER","BLISTER","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","PRURITUS","MALIGNANT FIBROUS HISTIOCYTOMA","MALIGNANT FIBROUS HISTIOCYTOMA","ERYTHEMA","PRURITUS","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","ERYTHEMA","PRURITUS","ERYTHEMA","PRURITUS","PRURITUS","SWELLING","BLISTER","HYPERSENSITIVITY","HYPERSENSITIVITY","ERYTHEMA","DELUSION","HALLUCINATION","PRURITUS","PROSTATE CANCER","PRURITUS","BLOOD CHOLESTEROL INCREASED","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","BLISTER","BLISTER","ERYTHEMA","ERYTHEMA","ERYTHEMA","ABDOMINAL PAIN","ULCER","PRURITUS","ERYTHEMA","COGNITIVE DISORDER","PRURITUS","ERYTHEMA","URTICARIA","URTICARIA","PRURITUS","HEADACHE","HEADACHE","VISION BLURRED","VISION BLURRED","HEADACHE","HEADACHE","ERYTHEMA","HYPERHIDROSIS","ERYTHEMA","PRURITUS","ELECTROCARDIOGRAM T WAVE INVERSION","CHEST PAIN","ENURESIS","DIARRHOEA","DIARRHOEA","ANXIETY","COUGH","COUGH","FATIGUE","IRRITABILITY","RHINORRHOEA","RHINORRHOEA","SKIN IRRITATION","SKIN IRRITATION","SKIN IRRITATION","SKIN IRRITATION","SINUS BRADYCARDIA","SKIN IRRITATION","BACK PAIN","WOUND","WOUND","INCREASED APPETITE","INCREASED APPETITE","HYPERHIDROSIS","SKIN IRRITATION","CONFUSIONAL STATE","INSOMNIA","PRURITUS GENERALISED","VENTRICULAR EXTRASYSTOLES","SKIN IRRITATION","SKIN IRRITATION","PRURITUS","PRURITUS","INSOMNIA","RASH","SKIN IRRITATION","ALOPECIA","FATIGUE","FATIGUE","NASAL CONGESTION","NAUSEA","VOMITING","DIZZINESS","SKIN IRRITATION","SKIN IRRITATION","OEDEMA PERIPHERAL","APPLICATION SITE PERSPIRATION","APPLICATION SITE PERSPIRATION","VOMITING","NAUSEA","DIZZINESS","ANXIETY","ANXIETY","SKIN IRRITATION","SINUS BRADYCARDIA","SINUS BRADYCARDIA","RASH","RASH","PAIN","PAIN","SYNCOPE","AGITATION","INFLAMMATION","SYNCOPE","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","CONFUSIONAL STATE","VOMITING","SINUS BRADYCARDIA","SINUS BRADYCARDIA","HYPERHIDROSIS","HYPERHIDROSIS","MYOCARDIAL INFARCTION","SOMNOLENCE","SOMNOLENCE","DIARRHOEA","BRADYCARDIA","BRADYCARDIA","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","APPLICATION SITE IRRITATION","BRADYCARDIA","BRADYCARDIA","FOOD CRAVING","APPLICATION SITE IRRITATION","SINUS BRADYCARDIA","DIZZINESS","BODY TEMPERATURE INCREASED","HYPERHIDROSIS","HYPERTENSION","SYNCOPE","AGITATION","BALANCE DISORDER","BALANCE DISORDER","BALANCE DISORDER","FALL","CONTUSION","CONTUSION","CONTUSION","COORDINATION ABNORMAL","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","ARTHRALGIA","ARTHRALGIA","ARTHRALGIA","APPLICATION SITE DISCOLOURATION","APPLICATION SITE PRURITUS","SOMNOLENCE","SOMNOLENCE","DYSPHAGIA","FOOD CRAVING","APPLICATION SITE DERMATITIS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","DERMATITIS ATOPIC","SKIN ULCER","SKIN ULCER","CONFUSIONAL STATE","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","SOMNOLENCE","APPLICATION SITE REACTION","APPLICATION SITE REACTION","HYPERTENSION","HYPERTENSION","FATIGUE","NAUSEA","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","HALLUCINATION, VISUAL","PARTIAL SEIZURES WITH SECONDARY GENERALISATION","BACK PAIN","NAUSEA","AMNESIA","AMNESIA","NAUSEA","LETHARGY","NAUSEA","NAUSEA","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE DERMATITIS","DECREASED APPETITE","DECREASED APPETITE","NAUSEA","NAUSEA"],["APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","DIARRHOEA","ERYTHEMA","ERYTHEMA","ERYTHEMA","ATRIOVENTRICULAR BLOCK SECOND DEGREE","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","FATIGUE","HIATUS HERNIA","HIATUS HERNIA","UPPER RESPIRATORY TRACT INFECTION","BUNDLE BRANCH BLOCK LEFT","ERYTHEMA","APPLICATION SITE VESICLES","PRURITUS GENERALISED","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","PRURITUS GENERALISED","PRURITUS GENERALISED","NASAL CONGESTION","PHARYNGOLARYNGEAL PAIN","PRURITUS GENERALISED","LOCALISED INFECTION","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","MICTURITION URGENCY","ARTHRALGIA","CELLULITIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","FATIGUE","APPLICATION SITE DERMATITIS","APPLICATION SITE IRRITATION","COUGH","PYREXIA","URINARY TRACT INFECTION","URINARY TRACT INFECTION","EYE ALLERGY","EYE SWELLING","EYE PRURITUS","NASAL CONGESTION","PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","RASH","RASH","APPLICATION SITE IRRITATION","FATIGUE","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PAIN","APPLICATION SITE IRRITATION","APPLICATION SITE VESICLES","DYSPEPSIA","DEPRESSED MOOD","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","LOWER RESPIRATORY TRACT INFECTION","LOWER RESPIRATORY TRACT INFECTION","FLANK PAIN","CALCULUS URETHRAL","EPISTAXIS","ACTINIC KERATOSIS","INCREASED APPETITE","INCREASED APPETITE","HEADACHE","VOMITING","VOMITING","MICTURITION URGENCY","MICTURITION URGENCY","NASOPHARYNGITIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","AGITATION","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","URTICARIA","URTICARIA","APPLICATION SITE URTICARIA","URTICARIA","COUGH","COUGH","NASAL MUCOSA BIOPSY","ERYTHEMA","ERYTHEMA","SECRETION DISCHARGE","SECRETION DISCHARGE","MUSCULAR WEAKNESS","MUSCULAR WEAKNESS","PNEUMONIA","PNEUMONIA","APPLICATION SITE ERYTHEMA","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE PRURITUS","EYE LASER SURGERY","APPLICATION SITE PRURITUS","JOINT DISLOCATION","SHOULDER PAIN","INCONTINENCE","APPLICATION SITE ERYTHEMA","SKIN LACERATION","CONFUSIONAL STATE","DYSPNOEA","SUDDEN DEATH","ERYTHEMA","ERYTHEMA","SKIN ODOUR ABNORMAL","PRURITUS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","FATIGUE","HORDEOLUM","NAUSEA","NAUSEA","VOMITING","ERYTHEMA","ERYTHEMA","PAROSMIA","PAROSMIA","PRURITUS","PRURITUS","SALIVARY HYPERSECRETION","SALIVARY HYPERSECRETION","COUGH","COUGH","NASAL CONGESTION","NASAL CONGESTION","SALIVARY HYPERSECRETION","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","RHINORRHOEA","RHINORRHOEA","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","RASH ERYTHEMATOUS","APPLICATION SITE PERSPIRATION","HYPERHIDROSIS","STOMACH DISCOMFORT","PAIN","NASAL CONGESTION","DIZZINESS","MALAISE","MALAISE","MYALGIA","PHARYNGOLARYNGEAL PAIN","PRODUCTIVE COUGH","DIZZINESS","CONTUSION","CONTUSION","HEADACHE","EPISTAXIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","RESPIRATORY TRACT CONGESTION","RHINITIS","FEELING ABNORMAL","LIBIDO DECREASED","LISTLESS","CYST","ONYCHOMYCOSIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE URTICARIA","DEPRESSED MOOD","DEPRESSED MOOD","FATIGUE","DYSPHONIA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","APPLICATION SITE VESICLES","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE IRRITATION","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","HEADACHE","HEADACHE","NAUSEA","APPLICATION SITE PRURITUS","BACK PAIN","BACK PAIN","APPLICATION SITE PAIN","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE VESICLES","CHEST DISCOMFORT","HEADACHE","COUGH","DIARRHOEA","HYPERHIDROSIS","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","MICTURITION URGENCY","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","DIARRHOEA","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","SALIVARY HYPERSECRETION","APPLICATION SITE IRRITATION","APPLICATION SITE VESICLES","PARAESTHESIA","NEUTROPHIL COUNT INCREASED","NEUTROPHIL COUNT INCREASED","URINE ANALYSIS ABNORMAL","URINE ANALYSIS ABNORMAL","WHITE BLOOD CELL COUNT INCREASED","WHITE BLOOD CELL COUNT INCREASED","RECTAL HAEMORRHAGE","RECTAL HAEMORRHAGE","APPLICATION SITE IRRITATION","SKIN IRRITATION","DIARRHOEA","INSOMNIA","HYPERCHOLESTEROLAEMIA","BIOPSY PROSTATE","BENIGN PROSTATIC HYPERPLASIA","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","HYPERHIDROSIS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","CHILLS","CHILLS","OEDEMA PERIPHERAL","OEDEMA PERIPHERAL","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","CATARACT OPERATION","UPPER RESPIRATORY TRACT INFECTION","OEDEMA PERIPHERAL","OEDEMA PERIPHERAL","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","PRURITUS","PRURITUS","ATRIOVENTRICULAR BLOCK SECOND DEGREE","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","SOMNOLENCE","SOMNOLENCE","DISORIENTATION","DECREASED APPETITE","DECREASED APPETITE","DIZZINESS","DIZZINESS","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","ARTHRITIS","GLAUCOMA","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","HYPOTENSION","HYPOTENSION","TACHYCARDIA","TACHYCARDIA","MYOCARDIAL INFARCTION","ATRIOVENTRICULAR BLOCK SECOND DEGREE","ATRIOVENTRICULAR BLOCK SECOND DEGREE","WEIGHT DECREASED","WEIGHT DECREASED","DIARRHOEA","VOMITING","CHEST DISCOMFORT","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","HYPERHIDROSIS","TRANSIENT ISCHAEMIC ATTACK","SINUS BRADYCARDIA","FATIGUE","MALAISE","MALAISE","RASH","DIZZINESS","NAUSEA","CYSTITIS","DIARRHOEA","VOMITING","ARTHRALGIA","CONTUSION","EXCORIATION","SKIN LACERATION","MYOCARDIAL INFARCTION","VENTRICULAR SEPTAL DEFECT","CARDIAC DISORDER","INFLUENZA","PRURITUS","PRURITUS","RASH","RASH","PALPITATIONS","VERTIGO","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","WOLFF-PARKINSON-WHITE SYNDROME","WOLFF-PARKINSON-WHITE SYNDROME","ABDOMINAL DISCOMFORT","APPLICATION SITE PRURITUS","RASH","APPLICATION SITE IRRITATION","HYPERHIDROSIS","MALAISE","CHILLS","DIZZINESS","DIARRHOEA","SINUS BRADYCARDIA","PRURITUS","ERYTHEMA","BLOOD GLUCOSE INCREASED","BLOOD GLUCOSE INCREASED","PRURITUS","RASH","ERYTHEMA","OEDEMA","PRURITUS","RASH","INFLUENZA","RASH","RASH","NASOPHARYNGITIS","NASOPHARYNGITIS","PRURITUS","PRURITUS","RASH","RASH","INFLUENZA","INFLUENZA","OEDEMA PERIPHERAL","BLOOD GLUCOSE INCREASED","RASH","DYSPNOEA","SEASONAL ALLERGY","ERYTHEMA","ERYTHEMA","ERYTHEMA","ATRIAL FLUTTER","ATRIAL FLUTTER","ATRIAL FIBRILLATION","ATRIAL FIBRILLATION","HYPERHIDROSIS","HYPERHIDROSIS","BLISTER","BLISTER","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","VIRAL INFECTION","OEDEMA","PRURITUS","RASH","RASH","PRURITUS","RASH","RASH","RASH","PRURITUS","PRURITUS","BURNING SENSATION","FLATULENCE","FLATULENCE","POLLAKIURIA","POLLAKIURIA","HEADACHE","PARKINSON'S DISEASE","COMPLETED SUICIDE","RASH","ERYTHEMA","HYPERBILIRUBINAEMIA","DYSPEPSIA","DYSPEPSIA","PRURITUS","EXCORIATION","PRURITUS","PRURITUS","PRURITUS","ERYTHEMA","PRURITUS","PRURITUS","RASH","RASH","RASH PRURITIC","WOUND HAEMORRHAGE","NAUSEA","VOMITING","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","VENTRICULAR HYPERTROPHY","PRURITUS","PRURITUS","ABDOMINAL PAIN","COLON CANCER","RASH ERYTHEMATOUS","ANXIETY","ANXIETY","ANXIETY","DIARRHOEA","EXCORIATION","EXCORIATION","IRRITABILITY","IRRITABILITY","PRURITUS","PRURITUS","SYNCOPE","SYNCOPE","ERYTHEMA","ERYTHEMA","EXCORIATION","EXCORIATION","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","SINUS ARRHYTHMIA","SINUS ARRHYTHMIA","BLOOD CREATINE PHOSPHOKINASE INCREASED","BLOOD CREATINE PHOSPHOKINASE INCREASED","POSTNASAL DRIP","POSTNASAL DRIP","CONJUNCTIVITIS","CONJUNCTIVITIS","HEADACHE","DIZZINESS","STUPOR","TRANSIENT ISCHAEMIC ATTACK","VISION BLURRED","APPLICATION SITE VESICLES","DIZZINESS","DIZZINESS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","SINUS BRADYCARDIA","SINUS BRADYCARDIA","HEART RATE IRREGULAR","HEART RATE IRREGULAR","DIZZINESS","HEADACHE","HEART RATE IRREGULAR","HEART RATE IRREGULAR","AGITATION","ERYTHEMA","FLANK PAIN","NASOPHARYNGITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","VENTRICULAR SEPTAL DEFECT","ALCOHOL USE","ATRIAL FIBRILLATION","CYSTITIS","DIZZINESS","ATRIAL FIBRILLATION","ATRIAL FIBRILLATION","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE SWELLING","CONJUNCTIVITIS","CONJUNCTIVITIS","COUGH","COUGH","HEART RATE INCREASED","HEART RATE INCREASED","DIZZINESS","DIZZINESS","DIARRHOEA","MUSCLE SPASMS","VENTRICULAR EXTRASYSTOLES","VENTRICULAR EXTRASYSTOLES","APPLICATION SITE DERMATITIS","COUGH","COUGH","NASOPHARYNGITIS","NASOPHARYNGITIS","HEADACHE","HEADACHE","HOT FLUSH","APPLICATION SITE PRURITUS","MYOCARDIAL INFARCTION","ABDOMINAL PAIN","CONSTIPATION","ATRIAL HYPERTROPHY","ATRIAL HYPERTROPHY","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","APPLICATION SITE VESICLES","PAIN IN EXTREMITY","APPLICATION SITE PRURITUS","APPLICATION SITE SWELLING","SINUS BRADYCARDIA","HEADACHE","APPLICATION SITE ERYTHEMA","DIZZINESS","APPLICATION SITE PRURITUS","DIZZINESS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE SWELLING","APPLICATION SITE SWELLING","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","APPLICATION SITE PRURITUS","DIZZINESS","DIZZINESS","HYPOTENSION","ATRIOVENTRICULAR BLOCK SECOND DEGREE","OEDEMA PERIPHERAL","OEDEMA PERIPHERAL","EPISTAXIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE WARMTH","APPLICATION SITE WARMTH","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","DIARRHOEA","APPLICATION SITE DERMATITIS","ABDOMINAL PAIN","DIARRHOEA","APPLICATION SITE PRURITUS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","APPLICATION SITE ERYTHEMA","MUSCLE SPASMS","MUSCLE SPASMS","MYOCARDIAL INFARCTION","MYOCARDIAL INFARCTION","SINUS BRADYCARDIA","SINUS BRADYCARDIA","COUGH","COUGH","PHARYNGEAL ERYTHEMA","PHARYNGEAL ERYTHEMA","MYOCARDIAL INFARCTION","SINUS BRADYCARDIA","PRURITUS","RASH","OEDEMA PERIPHERAL","APPLICATION SITE DERMATITIS","RASH","NASOPHARYNGITIS","NASOPHARYNGITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE VESICLES","ACROCHORDON EXCISION","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","DIZZINESS","APPLICATION SITE VESICLES","SKIN IRRITATION","SKIN IRRITATION","PRURITUS","PRURITUS","RASH","RASH","DIZZINESS","COUGH","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","DIZZINESS","ELECTROCARDIOGRAM T WAVE INVERSION","DIZZINESS","DIZZINESS","HEADACHE","PRURITUS","DIARRHOEA","APPLICATION SITE DERMATITIS","APPLICATION SITE DISCHARGE","APPLICATION SITE PRURITUS","HIP FRACTURE","COUGH","UPPER RESPIRATORY TRACT INFECTION","SKIN LESION EXCISION","LOCALISED INFECTION","LOCALISED INFECTION","BLOOD URINE PRESENT","BENIGN PROSTATIC HYPERPLASIA","BENIGN PROSTATIC HYPERPLASIA","CYSTOSCOPY","DEHYDRATION","HYPOTENSION","PYREXIA","ORTHOSTATIC HYPOTENSION","ORTHOSTATIC HYPOTENSION","HEMIANOPIA HOMONYMOUS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","SINUS BRADYCARDIA","DIABETES MELLITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","RASH","RASH","RASH","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","DRUG ERUPTION","RASH MACULO-PAPULAR","RASH MACULO-PAPULAR","RASH PAPULAR","RASH PAPULAR","NASAL CONGESTION","NIGHTMARE","VOMITING","VOMITING","RASH MACULO-PAPULAR","BIOPSY","DIZZINESS","DIZZINESS","ALLERGIC GRANULOMATOUS ANGIITIS","APPLICATION SITE PRURITUS","FEELING COLD","APPLICATION SITE DERMATITIS","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","SYNCOPE","VOMITING","APPLICATION SITE BLEEDING","COUGH","DIZZINESS","PYREXIA","HEADACHE","BLOOD ALKALINE PHOSPHATASE INCREASED","SYNCOPE","TRANSIENT ISCHAEMIC ATTACK","TRANSIENT ISCHAEMIC ATTACK","DIZZINESS","SALIVARY HYPERSECRETION","SKIN IRRITATION","DIZZINESS","NEPHROLITHIASIS","URINARY TRACT INFECTION","PRURITUS","SINUS BRADYCARDIA","SINUS BRADYCARDIA","CONTUSION","FACIAL BONES FRACTURE","FALL","CHEST PAIN","PRURITUS","PRURITUS","HAEMOPTYSIS","HAEMOPTYSIS","RALES","RALES","VOMITING","NASOPHARYNGITIS","NASOPHARYNGITIS","PRURITUS","PRURITUS","SKIN IRRITATION","SKIN IRRITATION","RASH","RASH","ERYTHEMA","ERYTHEMA","SKIN EXFOLIATION","SKIN EXFOLIATION","PRURITUS","PRURITUS","RASH","RASH","NASAL CONGESTION","INSOMNIA","INSOMNIA","APPLICATION SITE INDURATION","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","FALL","FALL","SUPRAVENTRICULAR EXTRASYSTOLES","ERYTHEMA","PRURITUS","RASH","HIP FRACTURE","GASTROINTESTINAL HAEMORRHAGE","ARTHRITIS","ARTHRITIS","APPLICATION SITE DERMATITIS","SKIN IRRITATION","AGITATION","MYOCARDIAL INFARCTION","ERYTHEMA","PRURITUS","VERTIGO","HEADACHE","DIZZINESS","HEADACHE","NAUSEA","MYOCARDIAL INFARCTION","SYNCOPE VASOVAGAL","SKIN IRRITATION","SKIN IRRITATION","COMPLEX PARTIAL SEIZURES","FALL","SKIN LACERATION","SYNCOPE","SYNCOPE","ATRIOVENTRICULAR BLOCK FIRST DEGREE","SINUS BRADYCARDIA","APPLICATION SITE IRRITATION","COUGH","SHOULDER PAIN","APPLICATION SITE PRURITUS","PRURITUS","PRURITUS","ERYTHEMA","PRURITUS","DIARRHOEA","PRURITUS","RASH","CHILLS","COLD SWEAT","CHILLS","COLD SWEAT","CHILLS","COLD SWEAT","ERYTHEMA","PRURITUS","ATRIAL FIBRILLATION","DYSPNOEA","MYOCARDIAL INFARCTION","CARDIAC FAILURE CONGESTIVE","HYPONATRAEMIA","SOMNOLENCE","RASH","RASH","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","BURNING SENSATION","VOMITING","PRURITUS","BLISTER","ERYTHEMA","APPLICATION SITE IRRITATION","NASOPHARYNGITIS","NASOPHARYNGITIS","HEADACHE","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","RASH","RASH","ERYTHEMA","PRURITUS","PRURITUS","RASH","APPLICATION SITE DESQUAMATION","HIP FRACTURE","HIP FRACTURE","DELUSION","PSYCHOMOTOR HYPERACTIVITY","SUPRAVENTRICULAR TACHYCARDIA","SUPRAVENTRICULAR TACHYCARDIA","ATRIOVENTRICULAR BLOCK FIRST DEGREE","PRURITUS","BLISTER","RASH","ATRIAL FIBRILLATION","ELECTROCARDIOGRAM T WAVE AMPLITUDE DECREASED","NASOPHARYNGITIS","NASOPHARYNGITIS","ERYTHEMA","ERYTHEMA","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","NASOPHARYNGITIS","APPLICATION SITE URTICARIA","HEADACHE","EAR INFECTION","EAR INFECTION","GLOSSITIS","TINNITUS","PARAESTHESIA ORAL","NASOPHARYNGITIS","NASOPHARYNGITIS","NASOPHARYNGITIS","SUPRAVENTRICULAR EXTRASYSTOLES","SUPRAVENTRICULAR EXTRASYSTOLES","APPLICATION SITE DERMATITIS","RASH PRURITIC","RASH PRURITIC","VENTRICULAR EXTRASYSTOLES","VENTRICULAR EXTRASYSTOLES","CONJUNCTIVAL HAEMORRHAGE","CERUMEN IMPACTION","HYPERTENSION","HYPERTENSION","ASTHENIA","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","NAUSEA","NAUSEA","VOMITING","VOMITING","CATARACT OPERATION","APPLICATION SITE PRURITUS","RASH PRURITIC","RASH PRURITIC","BACK PAIN","DELIRIUM","DYSURIA","DYSPEPSIA","URINARY TRACT INFECTION","URINARY TRACT INFECTION","ASTHENIA","ASTHENIA","DYSPEPSIA","GASTROENTERITIS VIRAL","GASTROOESOPHAGEAL REFLUX DISEASE","PELVIC PAIN","PELVIC PAIN","CERVICITIS","CERVICITIS","VAGINAL MYCOSIS","VAGINAL MYCOSIS","ABDOMINAL PAIN","ABDOMINAL PAIN","APPLICATION SITE PRURITUS","DIZZINESS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE ERYTHEMA","COUGH","SHOULDER PAIN","SHOULDER PAIN","RASH","RASH","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","NAUSEA","NAUSEA","BUNDLE BRANCH BLOCK RIGHT","BUNDLE BRANCH BLOCK RIGHT","ELECTROCARDIOGRAM T WAVE INVERSION","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","SUPRAVENTRICULAR EXTRASYSTOLES","SUPRAVENTRICULAR EXTRASYSTOLES","BRONCHITIS","SKIN IRRITATION","SKIN IRRITATION","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","DIARRHOEA","DIARRHOEA","HYPERHIDROSIS","HYPERHIDROSIS","APPLICATION SITE ERYTHEMA","APPLICATION SITE ERYTHEMA","SOMNOLENCE","APPLICATION SITE PRURITUS","DYSURIA","DIZZINESS","DIARRHOEA","ERYTHEMA","COUGH","NASOPHARYNGITIS","BACK PAIN","BACK PAIN","ARTHRALGIA","SINUS BRADYCARDIA","SKIN IRRITATION","SKIN IRRITATION","EAR PAIN","EAR PAIN","PRURITUS","PRURITUS","ERYTHEMA","ERYTHEMA","ATRIOVENTRICULAR BLOCK SECOND DEGREE","DERMATITIS CONTACT","DERMATITIS CONTACT","SINUS BRADYCARDIA","NEPHROLITHIASIS","SINUS BRADYCARDIA","NAUSEA","NAUSEA","APPLICATION SITE REACTION","SINUS BRADYCARDIA","CONFUSIONAL STATE","DIZZINESS","HYPERHIDROSIS","PALPITATIONS","SINUS BRADYCARDIA","SINUS BRADYCARDIA","NASOPHARYNGITIS","NASOPHARYNGITIS","EMPHYSEMA","HYPERHIDROSIS","PRURITUS","ELECTROCARDIOGRAM T WAVE INVERSION","ELECTROCARDIOGRAM T WAVE INVERSION","EAR INFECTION","ELECTROCARDIOGRAM T WAVE AMPLITUDE DECREASED","EAR INFECTION","DIARRHOEA","DIARRHOEA","CONFUSIONAL STATE","HYPERHIDROSIS","FATIGUE","PRURITUS","PRURITUS","RESTLESSNESS","RESTLESSNESS","LETHARGY","FATIGUE","HYPERSOMNIA","SYNCOPE","ATRIAL FLUTTER","BUNDLE BRANCH BLOCK RIGHT","VENTRICULAR SEPTAL DEFECT","BLISTER","ERYTHEMA","PRURITUS","BLISTER","BLISTER","ELECTROCARDIOGRAM ST SEGMENT DEPRESSION","PRURITUS","MALIGNANT FIBROUS HISTIOCYTOMA","MALIGNANT FIBROUS HISTIOCYTOMA","ERYTHEMA","PRURITUS","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","ERYTHEMA","PRURITUS","ERYTHEMA","PRURITUS","PRURITUS","SWELLING","BLISTER","HYPERSENSITIVITY","HYPERSENSITIVITY","ERYTHEMA","DELUSION","HALLUCINATION","PRURITUS","PROSTATE CANCER","PRURITUS","BLOOD CHOLESTEROL INCREASED","ERYTHEMA","ERYTHEMA","PRURITUS","PRURITUS","BLISTER","BLISTER","ERYTHEMA","ERYTHEMA","ERYTHEMA","ABDOMINAL PAIN","ULCER","PRURITUS","ERYTHEMA","COGNITIVE DISORDER","PRURITUS","ERYTHEMA","URTICARIA","URTICARIA","PRURITUS","HEADACHE","HEADACHE","VISION BLURRED","VISION BLURRED","HEADACHE","HEADACHE","ERYTHEMA","HYPERHIDROSIS","ERYTHEMA","PRURITUS","ELECTROCARDIOGRAM T WAVE INVERSION","CHEST PAIN","ENURESIS","DIARRHOEA","DIARRHOEA","ANXIETY","COUGH","COUGH","FATIGUE","IRRITABILITY","RHINORRHOEA","RHINORRHOEA","SKIN IRRITATION","SKIN IRRITATION","SKIN IRRITATION","SKIN IRRITATION","SINUS BRADYCARDIA","SKIN IRRITATION","BACK PAIN","WOUND","WOUND","INCREASED APPETITE","INCREASED APPETITE","HYPERHIDROSIS","SKIN IRRITATION","CONFUSIONAL STATE","INSOMNIA","PRURITUS GENERALISED","VENTRICULAR EXTRASYSTOLES","SKIN IRRITATION","SKIN IRRITATION","PRURITUS","PRURITUS","INSOMNIA","RASH","SKIN IRRITATION","ALOPECIA","FATIGUE","FATIGUE","NASAL CONGESTION","NAUSEA","VOMITING","DIZZINESS","SKIN IRRITATION","SKIN IRRITATION","OEDEMA PERIPHERAL","APPLICATION SITE PERSPIRATION","APPLICATION SITE PERSPIRATION","VOMITING","NAUSEA","DIZZINESS","ANXIETY","ANXIETY","SKIN IRRITATION","SINUS BRADYCARDIA","SINUS BRADYCARDIA","RASH","RASH","PAIN","PAIN","SYNCOPE","AGITATION","INFLAMMATION","SYNCOPE","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","CONFUSIONAL STATE","VOMITING","SINUS BRADYCARDIA","SINUS BRADYCARDIA","HYPERHIDROSIS","HYPERHIDROSIS","MYOCARDIAL INFARCTION","SOMNOLENCE","SOMNOLENCE","DIARRHOEA","BRADYCARDIA","BRADYCARDIA","UPPER RESPIRATORY TRACT INFECTION","UPPER RESPIRATORY TRACT INFECTION","APPLICATION SITE IRRITATION","BRADYCARDIA","BRADYCARDIA","FOOD CRAVING","APPLICATION SITE IRRITATION","SINUS BRADYCARDIA","DIZZINESS","BODY TEMPERATURE INCREASED","HYPERHIDROSIS","HYPERTENSION","SYNCOPE","AGITATION","BALANCE DISORDER","BALANCE DISORDER","BALANCE DISORDER","FALL","CONTUSION","CONTUSION","CONTUSION","COORDINATION ABNORMAL","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","ARTHRALGIA","ARTHRALGIA","ARTHRALGIA","APPLICATION SITE DISCOLOURATION","APPLICATION SITE PRURITUS","SOMNOLENCE","SOMNOLENCE","DYSPHAGIA","FOOD CRAVING","APPLICATION SITE DERMATITIS","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","DERMATITIS ATOPIC","SKIN ULCER","SKIN ULCER","CONFUSIONAL STATE","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","APPLICATION SITE DERMATITIS","SOMNOLENCE","APPLICATION SITE REACTION","APPLICATION SITE REACTION","HYPERTENSION","HYPERTENSION","FATIGUE","NAUSEA","APPLICATION SITE IRRITATION","APPLICATION SITE IRRITATION","HALLUCINATION, VISUAL","PARTIAL SEIZURES WITH SECONDARY GENERALISATION","BACK PAIN","NAUSEA","AMNESIA","AMNESIA","NAUSEA","LETHARGY","NAUSEA","NAUSEA","APPLICATION SITE DERMATITIS","APPLICATION SITE PRURITUS","APPLICATION SITE PRURITUS","APPLICATION SITE DERMATITIS","DECREASED APPETITE","DECREASED APPETITE","NAUSEA","NAUSEA"],["MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","SEVERE","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MODERATE","SEVERE","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MODERATE","SEVERE","MODERATE","SEVERE","SEVERE","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MILD","SEVERE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MILD","SEVERE","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","SEVERE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","SEVERE","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MODERATE","SEVERE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","SEVERE","MILD","SEVERE","SEVERE","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","SEVERE","MILD","SEVERE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","SEVERE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","SEVERE","MODERATE","SEVERE","MODERATE","SEVERE","MILD","MILD","SEVERE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","SEVERE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","SEVERE","SEVERE","SEVERE","MODERATE","MILD","MILD","MILD","SEVERE","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","SEVERE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","SEVERE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","SEVERE","SEVERE","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","SEVERE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","SEVERE","SEVERE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MODERATE","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","SEVERE","SEVERE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MILD","SEVERE","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","SEVERE","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MODERATE","SEVERE","MODERATE","MODERATE","SEVERE","MODERATE","SEVERE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MILD","SEVERE","SEVERE","MODERATE","MILD","MILD","MILD","MODERATE","MILD","MILD","MILD","MODERATE","MILD","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MILD","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MILD","MILD","MILD","MODERATE","MODERATE","SEVERE","MILD","MODERATE","MILD","MILD","MODERATE","MODERATE","MODERATE","SEVERE","MODERATE","MODERATE","MODERATE","MILD","MODERATE","MODERATE","MODERATE","MODERATE"],["N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","Y","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","Y","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","Y","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N","N"],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],["PROBABLE","PROBABLE","REMOTE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","POSSIBLE","NONE","NONE","NONE","NONE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","POSSIBLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","REMOTE","REMOTE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","POSSIBLE","NONE","NONE","NONE","POSSIBLE","NONE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","REMOTE","POSSIBLE","PROBABLE","PROBABLE","NONE","NONE","REMOTE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","NONE","NONE","POSSIBLE","PROBABLE","NONE","NONE","NONE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","REMOTE","REMOTE","REMOTE","POSSIBLE","NONE","NONE","POSSIBLE","REMOTE","REMOTE","NONE","REMOTE","PROBABLE","PROBABLE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","REMOTE","REMOTE","REMOTE","REMOTE","NONE","NONE","NONE","REMOTE","REMOTE","NONE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","POSSIBLE","REMOTE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","PROBABLE","PROBABLE","PROBABLE","NONE","REMOTE","REMOTE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","REMOTE","REMOTE","REMOTE","REMOTE","REMOTE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","NONE","NONE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","POSSIBLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","NONE","PROBABLE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE",null,null,"NONE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","POSSIBLE","PROBABLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","REMOTE","PROBABLE","PROBABLE","REMOTE","NONE","PROBABLE","POSSIBLE","REMOTE","REMOTE","REMOTE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","POSSIBLE","POSSIBLE","REMOTE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","NONE","NONE","POSSIBLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","REMOTE","REMOTE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","PROBABLE","REMOTE","REMOTE","REMOTE","REMOTE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","REMOTE","PROBABLE","PROBABLE","REMOTE","POSSIBLE","PROBABLE","POSSIBLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","REMOTE","REMOTE","POSSIBLE","NONE","NONE","NONE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","NONE","NONE","PROBABLE","REMOTE","REMOTE","NONE","NONE","REMOTE","REMOTE","NONE","NONE","NONE","NONE","REMOTE","REMOTE","PROBABLE","PROBABLE","REMOTE","PROBABLE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","NONE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","NONE","POSSIBLE","POSSIBLE","REMOTE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","NONE","POSSIBLE","NONE","POSSIBLE","POSSIBLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","PROBABLE","NONE","NONE","NONE","NONE","PROBABLE","NONE","PROBABLE","REMOTE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","NONE","POSSIBLE","REMOTE","NONE","NONE","POSSIBLE","REMOTE","REMOTE","NONE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","POSSIBLE","REMOTE","REMOTE","NONE","NONE","NONE","REMOTE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","NONE","REMOTE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","NONE","REMOTE","REMOTE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","REMOTE","NONE","NONE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","NONE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","REMOTE","NONE","NONE","NONE","REMOTE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","NONE","PROBABLE","PROBABLE","PROBABLE","REMOTE","POSSIBLE","REMOTE","POSSIBLE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","REMOTE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","REMOTE","REMOTE","REMOTE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","NONE","PROBABLE","REMOTE","POSSIBLE","POSSIBLE","PROBABLE","NONE","NONE","NONE","NONE","NONE","REMOTE","NONE","NONE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","NONE","NONE","NONE","REMOTE","REMOTE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","NONE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","NONE","REMOTE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","POSSIBLE","NONE","NONE","PROBABLE","NONE","NONE","POSSIBLE","NONE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","POSSIBLE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","NONE","PROBABLE","PROBABLE","REMOTE","REMOTE","POSSIBLE","PROBABLE","REMOTE","POSSIBLE","PROBABLE","NONE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","REMOTE","PROBABLE","REMOTE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","NONE","PROBABLE","PROBABLE","POSSIBLE","PROBABLE","PROBABLE","REMOTE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","REMOTE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","REMOTE","POSSIBLE","PROBABLE","PROBABLE","POSSIBLE","NONE","NONE","NONE","POSSIBLE","POSSIBLE","NONE","POSSIBLE","POSSIBLE","NONE","POSSIBLE","POSSIBLE","NONE","NONE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE","REMOTE","REMOTE","REMOTE","REMOTE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","NONE","NONE","NONE","PROBABLE","PROBABLE","REMOTE","REMOTE",null,null,"PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","REMOTE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","NONE","PROBABLE","PROBABLE","REMOTE","REMOTE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","POSSIBLE","NONE","NONE","REMOTE","NONE","NONE","POSSIBLE","POSSIBLE","POSSIBLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","PROBABLE","POSSIBLE","POSSIBLE","POSSIBLE","POSSIBLE"],["NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","FATAL","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","FATAL","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","FATAL","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED","NOT RECOVERED/NOT RESOLVED","RECOVERED/RESOLVED"],["2014-01-03","2014-01-03","2014-01-09","2012-08-07","2012-08-07","2012-08-07","2012-08-26","2013-07-21","2013-08-08","2014-08-27","2014-11-02","2013-02-12","2013-02-12","2013-03-06","2013-03-10","2014-01-03","2014-02-20","2014-02-20","2014-02-21","2014-02-21","2014-03-21","2014-03-31","2014-04-19","2014-04-19","2014-04-19","2012-07-08","2012-09-02","2012-09-02","2012-09-02","2012-09-02","2012-09-07","2012-09-13","2012-09-13","2012-12-02","2012-12-02","2012-12-02","2012-12-02","2012-12-26","2012-12-26","2013-01-14","2013-01-19","2013-01-19","2003","2014-03-09","2014-03-09","2014-03-09","2014-05-09","2014-05-09","2014-05-22","2014-05-22","2014-05-22","2012-12-27","2012-12-27","2012-12-27","2012-12-27","2013-05-16","2013-05-16","2013-06-01","2013-06-03","2013-06-10","2013-06-10","2013-06-10","2013-06-10","2013-06-22","2013-06-26","2013-06-26","2012-02","2013-07-29","2013-08-25","2013-08-25","2013-10-12","2013-10-12","2013-12-15","2013-12-17","2014-01-03","2014-02-12","2013-10-13","2013-10-13","2002","2013-02-12","2013-02-12","2013-02-15","2013-02-15","2013-03-06","2013-03-19","2013-03-19","2013-03-19","2013-12-09","2013-02-16","2013-02-16","2013-02-16","2013-02-16","2013-03-04","2013-03-04","2013-03-21","2013-03-21","2010-06","2010-06","2012-08-03","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-12-05","2012-12-05","2012-12-05","2012-12-05","2013-03-21","2012-11-16","2012-11-21","2012-11-21","2012-11-22","2012-11-29","2012-12-09","2013-01-11","2013-01-11","2013-01-14","2014-01-12","2014-01-12","2014-01-12","2014-01-13","2014-01-20","2014-01-20","2014-02-20","2014-02-20","2014-03","2014-04","2014-02-13","2014-02-13","2014-02-23","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-03-12","2014-03-12","2014-03-12","2014-03-12","2014-01-26","2014-02-22","2014-03-09","2014-07-09","2014-07-09","2013-03-22","2013-05-19","2013-05-19","2013-05-19","2013-05-19","2013-06-10","2013-08-30","2013-08-30","2013-08-30","2013-08-31","2013-09-01","2013-09-02","2013-09-02","2013-09-02","2013-09-02","2013-09-02","2013-09-02","2013-09-04","2013-09-05","2013-09-05","2013-09-15","2013-09-30","2013-10-15","2013-10-15","2013-10-23","2013-10-23","2013-10-24","2013-10-24","2013-10-24","2014-05-13","2014-05-13","2014-06-23","2014-06-23","2014-06-23","2014-07-04","2014-07-04","2014-09-12","2014-11-03","2012-10-22","2012-12-02","2012-12-25","2012-12-25","2013-01-19","2013-01-19","2013-01-26","2013-01-26","2013-01-26","2013-08-02","2013-08-02","2013-08-05","1986","1986","2013-06-14","2013-07-16","2013-10-13","2013-10-13","2013-02-07","2013-02-07","2013-03-23","2013-03-23","2013-03-23","2013-04-12","2013-04-12","2013-04-12","2013-05-07","2013-06-24","2013-06-24","2013-07-17","2014-03-18","2014-03-18","2013-03-16","2013-03-16","2013-10-21","2013-10-21","2013-12-02","2014-01-21","2014-01-21","2014-03-09","2014-01-10","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-02-04","2013-02-04","2013-02-08","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-09-02","2013-09-02","2013-09-09","2013-10-12","2013-03-04","2013-03-05","2007","2013-11-16","2013-11-20","2013-11-23","2013-11-23","2013-11-23","2013-11-25","2012-09-13","2012-09-13","2012-09-13","2012-09-13","2012-10-09","2012-10-09","2013-02-28","2013-02-28","2013-04-05","2013-04-05","2013-04-05","2013-04-25","2013-05-27","2013-07-17","2013-07-17","2013-03-16","2013-03-16","2013-05-04","2013-05-04","2013-12-18","2013-10-31","2013-10-31","2013-11-22","2013-11-22","2013-04-02","2013-05-04","2013-05-04","2001","2001","2012-08-11","2012-08-11","2012-09-04","2012-09-04","2012-09-04","2013-12-28","1992","2012-09-27","2012-09-27","2013-03-01","2013-03-01","2013-03-01","2013-03-01","2013-03-13","2014-03-15","2014-03-15","2012-12-12","2012-12-12","2012-12-13","2012-12-13","2012-12-15","2014-04-24","2014-04-24","2013-01-30","2013-02-21","2013-02-25","2013-09-05","2013-09-05","2013-09-05","2013-09-14","2013-09-17","2013-09-22","2014-02-27","2014-07-05","2014-07-05","2014-07-06","2014-07-06","2014-07-06","2014-07-06","2013-10-19","2013-10-19","2013-11-01","2013-11-01","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-09-28","2013-09-28","2013-10-11","2013-10-11","2013-10-25","2013-10-25","2013-10-26","2013-10-27","2013-10-27","2013-11-09","2013-11-09","2013-11-09","2013-11-17","2013-11-17","2013-12-15","2013-12-24","2014-01-22","2014-02-09","2013-03-30","2013-03-30","2013-05-04","2013-05-04","2013-02-21","2013-02-23","2013-03-23","2013-03-23","2013-06-07","2013-12-23","2013-12-23","2013-12-28","2013-12-28","2014-01-27","2014-01-27","2013-12-08","2013-12-08","2012-09-27","2012-09-27","2013-02-04","2012-12-30","2013-08-14","2013-09-28","2013-10-06","2013-10-17","2013-10-17","2013-10-17","2013-10-27","2013-10-27","2013-11-08","2013-11-08","2013-11-09","2013-11-09","2013-11-15","2013-11-15","2013-11-15","2013-11-15","2013-12-06","2013-12-06","2013-07-30","2013-08-04","2013-08-04","2013-08-04","2014-05-23","2014-07-04","2013-12-10","2013-12-10","2013-12-10","2014-01-18","2014-01-18","2014-02-03","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2012-08-23","2013-01-06","2014-10-31","2013-12-25","2013-11-02","2014-01-26","2013-09-17","2013-09-17","2014-04-20","2014-01-30","2014-01-30","2014-01-30","2013-10-28","2014-02-13","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2013-11-14","2014-01-09","2013-04-08","2013-04-11","2013-08-25","2013-08-25","2011-12-05","2011-12-05","2012-09-19","2013-01-21","2013-06-22","2013-07-04","2012-05","2012-05","2014-01-07","2014-01-14","2014-01-14","2014-01-15","2014-01-15","2014-03-04","2014-03-04","2013-06-03","2013-06-18","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-23","2012-09-23","2012-09-23","2012-09-23","2013-11-25","2013-11-25","2013-11-26","2013-11-26","2013-12-03","2013-12-03","2014-02-08","2014-02-08","2013-12-31","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2013-02-26","2013-05-23","2013-05-23","2013-06-09","2013-06-09","2013-06-23","2013-06-23","2013-01-19","2013-01-19","2013-02-04","2013-02-16","2013-03-16","2013-03-16","2014-02-25","2014-03-22","2013-12-19","2014-03-01","2014-03-09","2014-03-09","2014-03-09","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2013-02-22","2013-02-22","2012-11-06","2012-11-06","2012-11-06","2012-11-22","2013-05-06","2013-05-06","2013-05-06","2013-05-06","2013-05-07","2013-05-07","2013-06-07","2013-06-07","2013-02-11","2013-02-18","2013-02-21","2013-02-21","2013-02-27","2013-03-01","2013-03-01","2013-03-01","2013-03-01","2013-03-07","2013-03-07","2013-03-19","2013-09-12","2013-09-23","2014-02-18","2014-02-18","2013-06-27","2013-06-27","2013-09-04","2013-09-04","2013-10-04","2013-10-04","2013-12-07","2013-04-02","2013-04-02","2013-05-08","2014-01-29","2014-02-15","2012-12-18","2013-01-12","2013-02-02","2013-04-10","2013-04-10","2013-04-10","2013-04-10","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-08-18","2013-08-18","2013-12-01","2013-08-18","2013-08-18","2013-08-28","2013-04-10","2014-06-16","2014-06-16","2013-11-22","2013-12-10","2013-12-10","2013-12-10","2013-12-10","2013-12-10","2013-12-10","2012-08-16","2012-08-16","2012-08-21","2012-08-30","2012-12-27","2013-01-03","2013-01-03","2013-01-15","2013-01-09","2013-01-09","2013-01-17","2013-01-24","2013-01-24","2013-03-20","2013-03-20","2013-03-20","2013-03-20","2013-05-12","2013-05-12","2013-05-12","2013-05-12","2013-05-15","2013-06-12","2013-06-19","2014-04-07","2014-05-01","2014-05-08","2014-08-02","2013-11-06","2013-11-06","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2014-03-23","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-03-20","2013-03-26","2013-08-17","2013-08-17","2013-09-06","2013-09-06","2013-09-06","2013-09-06","2013-03-06","2013-03-20","2013-03-26","2013-03-26","2013-03-26","2013-03-27","2013-03-30","2013-04-01","2013-04-01","2013-05-17","2013-05-27","2013-05-28","2013-05-30","2013-05-30","2013-05-30","2013-05-31","2013-06-09","2013-06-09","2013-06-28","2013-02-05","2013-02-05","2013-03-14","2013-03-17","2013-03-17","2013-03-17","2013-04-13","2013-04-13","2013-04-13","2013-04-20","2013-04-20","2013-04-14","2013-04-19","2013-04-19","2013-04-19","2013-04-19","2013-05-19","2013-08-30","2014-04-20","2014-04-20","2014-04-20","2014-04-20","2014-04-20","2014-04-20","2014-05-04","2014-05-04","2014-05-20","2013-06-10","2013-06-10","2013-06-18","2013-06-18","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-20","2013-06-26","2013-07-01","2013-07-01","2013-07-04","2013-07-28","2013-09-06","2014-05-17","2014-10-19","2013-05-01","2013-05-01","2013-05-21","2013-05-21","2013-06-07","2013-07-12","2013-08-25","2013-08-25","2011-11","2013-05-12","2013-03-07","2014-01-16","2014-01-16","2013-02-18","2013-03-01","2013-03-01","2013-04-16","2013-04-17","2013-04-17","2013-05-18","2013-06-29","2013-06-29","2013-08-08","2013-08-08","2013-08-08","2013-10-10","2013-10-21","2013-10-21","2014-05-09","2014-05-09","2014-05-23","2014-05-23","2013-06-22","2013-07-02","2013-07-02","2013-07-02","2013-07-02","2013-07-02","2013-07-02","2013-07-22","2013-07-22","2013-08-03","2013-08-03","2013-08-13","2013-08-13","2013-01-28","2013-01-28","2013-01-28","2013-01-28","2013-01-02","2013-01-04","2013-01-04","2013-01-21","2013-01-23","2013-01-31","2013-02-08","2013-02-09","2012-10-06","2012-12-12","2012-12-12","2012-12-12","2013-01-22","2013-02-05","1977","1977","2013-12-12","2013-12-27","2014-01-12","2013-08-02","2013-10-13","2013-10-13","2013-11-04","2013-11-13","2012-10-04","2012-10-04","2012-10-14","2012-10-19","2012-10-19","2014-04-18","2014-04-18","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-20","2014-03-10","2014-03-10","2012-12-23","2012-11-22","2013-01-13","2013-01-29","2014-02-11","2014-02-18","2014-02-23","2014-04-24","2013-07-03","2013-08-21","2013-08-21","2013-09-29","2013-09-29","2013-10-09","2013-10-09","2013-10-11","2013-10-11","2014-02-23","2014-02-23","2012-11-21","2012-11-21","2012-11-21","2012-11-22","2012-11-22","2013-01-13","2013-02-07","2013-02-07","2013-02-24","2013-02-24","2013-02-24","2013-03-02","2013-01-31","2013-02-10","2013-02-10","2013-02-13","2013-05-20","2013-05-20","2013-02-26","2013-05-14","2013-05-14","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2012-12-30","2013-02-14","2012-12-22","2012-12-22","2013-01-24","2013-12-25","2013-12-25","2014-03-26","2014-03-26","2012-11-12","2012-11-12","2012-11-26","2012-12-18","2012-12-27","2012-12-27","2013-02-18","2013-02-18","2013-01-31","2013-01-31","2013-02-24","2013-02-24","2013-04-16","2013-04-16","2013-04-16","2013-04-16","2013-04-16","2013-04-27","2013-04-29","2012-07-29","2012-08-04","2012-08-04","2012-08-12","2007-10","2013-04-11","2013-04-28","2013-04-28","2013-04-28","2013-04-29","2013-04-29","2013-05-18","2013-05-27","2013-05-27","2013-05-28","2013-05-28","2013-05-30","2013-06-01","2012-07-23","2012-07-23","2013-02-05","2013-09-26","2013-09-26","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2014-03-08","2014-07-14","2013-06-21","2013-06-21","2013-06-30","2013-07-01","2013-08-19","2013-08-22","2013-08-28","2013-08-28","2013-09-05","2013-09-05","2013-09-12","2013-10-10","2013-10-10","2013-10-10","2013-10-10","2013-11-23","2013-11-23","2013-11-23","2013-11-23","2013-05-30","2013-05-30","2013-06-02","2013-10-31","2012-10-30","2012-10-30","2014-02-02","2014-04-18","2014-06-01","2014-06-01","2014-07-10","2014-07-10","2014-03-19","2014-03-19","2014-03-27","2014-03-27","2014-05-26","2014-05-26","2014-05-26","2014-06-04","2014-06-04","2014-06-04","2014-07-18","2013-08-12","2013-08-12","2013-09-20","2013-09-20","2013-09-20","2013-09-20","2013-05-12","2013-05-12","2013-05-12","2013-05-12","2013-05-13","2013-05-13","2013-05-14","2013-05-17","2013-07-25","2013-12-13","2013-12-14","2014-01-11","2014-01-22","2014-01-22","2014-02-24","2014-02-24","2014-03-08","2014-05-27","2013-03-30","2013-03-30","2013-04-04","2013-04-04","2013-05-29","2013-05-29","2013-06-16","2013-06-16","2013-08-13","2013-05-07","2013-05-07","2013-05-13","2013-01-10","2013-03-18","2014-03-14","2014-03-14","2014-04-21","2014-04-22","2013-07-08","2013-07-08","2013-07-08","2013-07-08","2013-07-13","2013-07-13","2012-08-20","2012-08-20","2012-12-16","2014-04-17","2014-04-17","2014-04-18","2014-04-18","2014-07-16","2014-07-31","2014-08-15","2013-05-21","2013-05-21","2013-10-15","2013-05-09","2013-07-01","2013-07-04","2013-07-04","2013-07-28","2013-07-28","2013-07-31","2013-07-13","2013-07-13","2013-07-31","2013-01-19","2013-01-19","2013-01-19","2014-07-09","2014-07-09","2014-07-09","2014-08-17","2014-08-17","2013-04-12","2013-05-01","2013-02-19","2013-02-19","2013-03-15","2013-03-15","2013-10-17","2013-10-17","2013-10-25","2013-10-25","2013-05-09","2013-05-09","2012-11-17","2012-11-17","2012-12-03","2013-03-10","2013-03-10","2014-08-24","2012-11-24","2012-11-24","2013-01-05","2013-02-07","2013-02-20","2013-03-06","2013-04-22","2013-04-22","2013-04-22","2013-04-22","2013-06-08","2013-06-08","2013-09-20","2014-05-28","2014-05-28","2014-07-22","2014-08-02","2013-08-18","2013-09-02","2012-12-14","2012-12-15","2013-05-05","2013-05-05","2013-05-05","2013-05-07","2013-07","2013-07","2013-07","2013-07","2013-09-26","2013-09-26","2014-05-11","2014-01-01","2014-01-14","2014-01-14","2014-03-14","2014-04-05","2013-05","2014-01-13","2014-01-13","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-03-09","2014-03-09","2014-04-08","2014-04-21","2014-05-21","2014-06-03","2014-06-03","2014-01-29","2014-01-29","2014-02-15","2014-02-15","2014-02-20","2014-02-22","2014-03-04","2014-04-18","2013-01-26","2013-01-26","2013-02-03","2013-02-03","2013-04-17","2013-04-25","2013-12-19","2014-01-15","2014-02-01","2014-02-01","2014-02-01","2014-02-28","2014-02-28","1994-04","2013-05-23","2013-05-23","2013-05-29","2013-06-11","2013-06-11","2013-07-07","2013-07-22","2013-08-10","2013-09-15","2013-09-15","2013-09-19","2013-10-06","2013-10-06","2013-10-07","2013-10-07","2014-01-13","2014-01-28","2013-07-16","2013-07-18","2013-07-30","2013-07-30","2012-10-05","2012-10-05","2012-10-05","2013-03-06","2013-04-13","2013-04-13","2013-04-14","2013-04-14","2013-07-06","2013-06-03","2013-06-03","2013-11-16","2013-02-16","2013-02-16","2013-03-06","2013-03-06","2013-03-15","2013-05-10","2013-05-10","2013-06-09","2013-06-16","2013-07-31","2013-10-07","2013-10-11","2013-10-11","2013-10-11","2013-10-12","2013-10-16","2013-10-09","2013-10-09","2013-10-09","2013-10-09","2013-10-15","2013-10-15","2013-10-15","2013-11-15","2013-12-29","2013-12-29","2013-12-29","2013-07-13","2013-07-25","2013-07-25","2013-08-13","2013-08-13","2013-09-03","2013-09-03","2013-10-31","2013-10-31","2013-03-23","2013-03-23","2013-04-02","2013-04-02","1982","2013-03-08","2013-03-08","2013-04-17","2013-04-27","2013-05-12","2013-05-16","2013-05-20","2013-05-25","2013-06-23","2013-06-23","2013-08-15","2013-08-15","2013-05-09","2013-05-09","2013-05-24","2013-05-24","2013-06-02","2013-06-02","2012-12-18","2012-12-26","2013-01-01","2013-01-01","2013-01-03","2013-01-05","2013-01-16","2013-01-21","2013-01-27","2013-01-27","2013-01-27","2013-01-29","2013-02-04","2013-02-04","2013-02-04","2013-02-04"],[null,null,"2014-01-11","2012-08-30",null,"2012-08-30",null,null,null,null,null,"2013-02-12","2013-02-12",null,null,null,"2014-02-22","2014-02-20",null,null,"2014-03-21","2014-03-31","2014-04-22","2014-04-22","2014-04-20",null,"2012-09-07","2012-09-07","2012-09-07","2012-09-07",null,null,null,null,null,null,null,"2012-12-26","2012-12-26",null,null,null,null,"2014-03-09","2014-03-16","2014-03-16","2014-05-09","2014-05-09",null,null,null,null,null,null,null,"2013-06-02","2013-06-02","2013-06-02",null,null,null,null,null,null,null,null,null,null,null,null,"2013-11-10","2013-11-10","2013-12-17","2013-12-17","2014-01-03",null,"2013-11-06","2013-11-06",null,"2013-02-12","2013-02-12","2013-03-19","2013-03-19","2013-03-07",null,null,null,"2013-12-09",null,null,null,null,null,null,null,null,null,null,"2012-08-03","2012-09-14","2012-09-14","2012-09-14","2012-09-14","2012-09-30","2012-09-30","2012-10-06","2012-10-06",null,"2012-12-16","2012-12-16",null,"2013-03-21",null,"2012-11-21",null,null,null,null,null,null,"2013-01-14",null,null,null,null,"2014-02-08","2014-02-08","2014-03-12","2014-03-12",null,null,"2014-06-03","2014-06-03","2014-02-23","2014-06-12","2014-06-12","2014-06-03","2014-06-03","2014-06-12","2014-06-12","2014-06-03","2014-06-03","2014-03-27","2014-03-27","2014-03-27","2014-03-27",null,null,null,"2014-07-12","2014-07-12","2013-03-23",null,null,null,null,null,null,null,null,null,null,"2013-09-03",null,null,null,null,null,null,"2013-10-08","2013-10-08",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-01-16","2013-01-16",null,null,null,"2013-01-30","2013-01-30",null,null,null,null,null,"2013-06-15",null,"2013-10-15","2013-10-15","2013-02-07","2013-02-07","2013-03-26",null,null,null,null,null,"2013-05-24","2013-06-24","2013-06-24","2013-07-26","2014-03-18","2014-03-18","2013-03-21","2013-03-21","2013-11-04","2013-11-04",null,"2014-02-10","2014-02-10","2014-03-09",null,null,null,null,null,null,null,null,"2013-02-08","2013-07-26","2013-07-26","2013-07-24","2013-07-24","2013-07-26","2013-07-26","2013-09-06","2013-09-06","2013-09-24","2013-10-31","2013-03-05","2013-03-06",null,"2013-11-16",null,null,null,null,null,"2013-01-02","2013-01-02","2013-01-02","2013-01-02","2012-10-15","2012-10-15","2013-09-14","2013-09-14","2013-04-16","2013-04-16","2013-04-16","2013-04-25","2013-06-03","2013-09-14","2013-09-14",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2012-08-19","2012-08-19","2013-01-25","2013-01-25","2013-01-25",null,null,"2012-10-06","2012-10-06",null,null,null,null,null,"2014-03-30","2014-03-30","2012-12-19","2012-12-19","2012-12-14","2012-12-14","2012-12-16",null,null,null,"2013-02-21",null,null,"2013-09-27","2013-09-27",null,null,"2013-09-23","2014-03-06","2014-07-06","2014-07-06",null,null,null,null,null,null,null,"2013-11-08","2013-11-22","2013-11-22","2013-11-22","2013-11-22","2013-09-30","2013-09-30","2013-10-25","2013-10-25","2013-11-29","2013-11-29",null,null,null,null,null,null,"2013-11-17","2013-11-17","2013-12-15",null,null,null,"2013-04-13","2013-04-13",null,null,"2013-02-23",null,null,null,"2013-06-13",null,null,"2014-02-03","2014-02-03",null,null,null,null,"2012-10-07","2012-10-07","2013-02-13",null,null,null,null,"2013-12-23","2013-12-23","2013-12-23","2013-11-08","2013-11-08","2013-12-06","2013-12-06","2013-12-04","2013-12-04","2013-12-14","2013-12-14","2013-12-23","2013-12-23","2013-12-23","2013-12-23","2013-08-03",null,null,null,null,null,"2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24",null,"2014-04-04","2014-04-04","2014-04-04","2014-04-04",null,null,"2014-10-31",null,null,null,"2013-09-29","2013-09-29",null,null,null,null,null,null,"2014-03-03","2014-03-03","2014-03-03","2014-03-03",null,null,"2013-04-08","2013-04-11",null,null,"2013-02-20","2013-02-20","2012-09-24","2013-01-24",null,"2013-07-06",null,null,"2014-01-08","2014-01-28","2014-01-28","2014-07-29","2014-07-29","2014-07-29","2014-07-29","2013-06-03","2013-06-18","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2013-12-23","2013-12-23","2013-12-09","2013-12-09","2013-12-10","2013-12-10","2014-02-22","2014-02-22","2013-12-31","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2013-03-09","2013-05-23","2013-05-23","2013-06-09","2013-06-09","2013-06-23","2013-06-23","2013-01-19","2013-01-19","2013-02-08",null,"2013-03-16","2013-03-16",null,"2014-03-22",null,"2014-03-08","2014-04-06","2014-04-06",null,"2014-03-24","2014-03-26","2014-03-28","2014-03-26","2013-02-22","2013-02-22","2012-12-13","2012-12-13",null,null,"2013-05-16","2013-05-16","2013-05-16","2013-05-16","2013-05-10","2013-05-10","2013-06-07","2013-06-07","2013-02-11","2013-02-18","2013-03-20","2013-03-20","2013-02-27","2013-03-07","2013-03-07","2013-03-07","2013-03-07","2013-03-07","2013-03-07","2013-03-19",null,null,"2014-02-18","2014-02-18","2013-06-27","2013-06-27","2013-09-18","2013-09-18","2013-10-30","2013-10-30","2013-12-07",null,null,null,null,null,"2012-12-18","2013-01-12","2013-02-02","2013-06-05","2013-06-05","2013-06-05","2013-06-05","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-08-31","2013-08-31",null,"2013-09-10","2013-09-10","2013-09-10",null,null,null,"2013-11-22","2013-12-21","2013-12-21","2013-12-21","2013-12-21","2013-12-21","2013-12-21","2012-08-19","2012-08-19",null,"2012-08-30",null,"2013-01-11","2013-01-11",null,"2013-02-19","2013-02-19",null,"2013-02-27","2013-02-27","2013-04-16","2013-04-16","2013-04-16","2013-04-16","2013-05-19","2013-05-19","2013-05-19","2013-05-19",null,null,null,null,null,null,null,"2013-11-17","2013-11-17",null,null,null,null,null,"2014-03-23",null,null,null,null,"2013-03-26","2013-03-27","2013-09-06","2013-09-06","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-03-06",null,"2013-06-18","2013-06-18","2013-06-18","2013-03-27",null,"2013-04-11","2013-04-11","2013-05-18","2013-06-18","2013-05-28",null,null,null,"2013-05-31",null,null,"2013-06-28","2013-02-13","2013-02-13",null,"2013-03-30","2013-03-30","2013-03-17","2013-04-14","2013-04-14","2013-04-14","2013-06-13","2013-06-13",null,"2013-05-31","2013-05-31","2013-05-31","2013-05-31",null,null,"2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-05-23","2013-07-01","2013-07-01","2013-07-16","2013-07-16","2013-06-20","2013-06-20","2013-06-19","2013-06-19",null,"2013-06-26","2013-07-01","2013-07-01","2013-07-04",null,null,null,null,null,null,"2013-05-21","2013-05-21","2013-06-17",null,"2013-08-25","2013-08-25",null,null,"2013-03-07","2014-01-20","2014-01-20","2013-02-19",null,null,"2013-04-16","2013-04-17","2013-04-19",null,"2013-07-04","2013-07-04",null,null,"2013-08-08","2013-10-10","2013-10-31","2013-10-31","2014-06-01","2014-06-01","2014-06-20","2014-06-20","2013-06-22","2013-07-09","2013-07-09","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-01-02","2013-01-15","2013-01-15","2013-01-21","2013-01-23","2013-01-31","2013-02-08","2013-02-09",null,null,null,null,"2013-01-25","2013-02-10",null,null,"2013-12-12",null,null,"2013-08-02",null,null,null,null,"2012-10-06","2012-10-06","2012-10-14","2012-10-19","2012-10-19",null,null,"2013-03-20","2013-03-19","2013-03-19","2013-03-19","2013-03-20",null,null,null,"2012-12-14",null,null,"2014-02-11","2014-02-18",null,null,"2013-07-05",null,null,"2013-09-29","2013-09-29","2013-10-09","2013-10-09","2013-10-11","2013-10-11",null,null,"2012-11-26","2012-11-21","2012-11-21",null,"2012-11-26","2013-01-19","2013-03-03","2013-03-03","2013-03-03","2013-03-03","2013-02-26","2013-03-02",null,null,null,null,"2013-05-24","2013-05-24","2013-02-26","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07",null,null,null,null,null,"2014-04-24","2014-04-24",null,null,"2012-11-13","2012-11-13",null,null,null,null,null,null,"2013-02-09","2013-02-09","2013-05-31","2013-05-31",null,null,"2013-05-02","2013-05-02","2013-05-02",null,"2013-05-01",null,"2012-08-12","2012-08-12",null,null,null,"2013-05-11","2013-05-11","2013-05-11","2013-05-15","2013-05-15",null,"2013-05-31","2013-05-31","2013-06-01","2013-06-01",null,null,null,null,null,null,null,"2013-10-11","2013-10-11","2013-10-11","2013-10-11","2014-03-08",null,"2013-07-15","2013-07-15",null,"2013-07-15","2013-08-23","2013-08-22","2013-09-08","2013-09-08",null,null,null,"2013-10-13",null,"2014-01-18","2014-01-18","2013-12-02","2013-12-02","2013-12-02","2013-12-02",null,null,null,null,"2012-10-31","2012-10-31",null,null,null,null,null,null,"2014-05-23","2014-05-23","2014-05-23","2014-05-23","2014-06-04","2014-06-04",null,null,"2014-06-17","2014-06-17","2014-07-24","2013-08-22","2013-08-22",null,null,null,null,"2013-07-17","2013-07-17","2013-07-17","2013-07-17","2013-07-17","2013-07-17",null,null,"2013-08-06","2013-12-13","2013-12-15",null,"2014-01-25","2014-01-25","2014-04-08","2014-04-08",null,null,"2013-04-06","2013-04-06","2013-04-15","2013-04-15",null,null,null,null,null,"2013-05-21","2013-05-21",null,"2013-01-10","2013-03-18","2014-03-14","2014-03-14",null,null,"2013-07-08","2013-07-08","2013-07-08","2013-07-08","2013-07-13","2013-07-13","2012-09-27","2012-09-27",null,null,null,"2014-07-31","2014-07-31","2014-07-25",null,"2014-08-27","2013-05-26","2013-05-26",null,null,null,"2013-07-28","2013-07-28","2013-07-31","2013-07-31",null,null,null,"2013-07-31",null,null,null,"2014-07-16",null,null,"2014-08-24","2014-08-24",null,null,"2013-03-22","2013-03-22",null,null,"2013-11-04","2013-11-04",null,null,null,null,null,null,null,"2013-04-10","2013-04-10",null,null,null,null,null,null,null,"2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06",null,"2014-09-22","2014-09-22","2014-08-02",null,null,null,null,null,null,"2013-05-07","2013-05-07",null,"2013-09-26","2013-09-26","2013-10-04","2013-10-04","2013-11-11","2013-11-11",null,null,null,null,null,"2014-04-09",null,"2014-02-10","2014-02-10",null,"2014-02-18","2014-02-18",null,null,"2014-02-08","2014-02-08","2014-02-09","2014-02-09","2014-03-15","2014-03-15",null,null,null,"2014-06-10","2014-06-10","2014-07-16","2014-07-16","2014-02-17",null,null,null,null,null,null,null,null,null,"2013-04-19","2013-04-26",null,null,"2014-02-18","2014-02-18","2014-02-06","2014-03-01","2014-03-01",null,null,null,null,"2013-10-15","2013-10-15","2013-07-08","2013-07-22","2013-08-10","2013-09-16","2013-09-16",null,"2014-03-03","2014-03-03","2013-10-11","2013-10-11","2014-01-14",null,"2013-07-20",null,null,null,null,null,null,"2013-03-06","2013-05-12","2013-05-12","2013-07-06","2013-07-06",null,"2013-07-16","2013-07-16","2013-11-16","2013-03-17","2013-03-17","2013-03-26","2013-03-26","2013-03-15","2013-07-07","2013-07-07",null,null,null,"2013-10-07","2013-10-12","2013-10-12","2013-10-12","2013-10-13",null,"2013-11-15","2013-11-15","2013-11-15","2013-10-09","2013-11-02","2013-11-02","2013-11-02","2013-11-15","2014-02-19","2014-02-19","2014-02-19",null,"2013-07-25","2013-07-25",null,null,"2013-09-19","2013-09-19",null,null,"2013-03-25","2013-03-25","2013-04-21","2013-04-21",null,"2013-03-22","2013-03-22",null,"2013-04-27","2013-05-12","2013-05-16","2013-05-20",null,"2013-08-13","2013-08-13","2013-08-29","2013-08-29","2013-05-11","2013-05-11",null,null,"2013-06-03","2013-06-05","2012-12-18","2012-12-26","2013-01-19","2013-01-19","2013-01-09","2013-01-05","2013-01-16","2013-01-24","2013-01-27","2013-01-28","2013-01-28",null,"2013-02-25","2013-02-25","2013-02-25","2013-02-25"],["2014-01-03","2014-01-03","2014-01-09","2012-08-07","2012-08-07","2012-08-07","2012-08-26","2013-07-21","2013-08-08","2014-08-27","2014-11-02","2013-02-12","2013-02-12","2013-03-06","2013-03-10","2014-01-03","2014-02-20","2014-02-20","2014-02-21","2014-02-21","2014-03-21","2014-03-31","2014-04-19","2014-04-19","2014-04-19","2012-07-08","2012-09-02","2012-09-02","2012-09-02","2012-09-02","2012-09-07","2012-09-13","2012-09-13","2012-12-02","2012-12-02","2012-12-02","2012-12-02","2012-12-26","2012-12-26","2013-01-14","2013-01-19","2013-01-19","2003-01-01","2014-03-09","2014-03-09","2014-03-09","2014-05-09","2014-05-09","2014-05-22","2014-05-22","2014-05-22","2012-12-27","2012-12-27","2012-12-27","2012-12-27","2013-05-16","2013-05-16","2013-06-01","2013-06-03","2013-06-10","2013-06-10","2013-06-10","2013-06-10","2013-06-22","2013-06-26","2013-06-26","2012-02-01","2013-07-29","2013-08-25","2013-08-25","2013-10-12","2013-10-12","2013-12-15","2013-12-17","2014-01-03","2014-02-12","2013-10-13","2013-10-13","2002-01-01","2013-02-12","2013-02-12","2013-02-15","2013-02-15","2013-03-06","2013-03-19","2013-03-19","2013-03-19","2013-12-09","2013-02-16","2013-02-16","2013-02-16","2013-02-16","2013-03-04","2013-03-04","2013-03-21","2013-03-21","2010-06-01","2010-06-01","2012-08-03","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-09-07","2012-09-07","2012-09-07","2012-09-07","2012-12-05","2012-12-05","2012-12-05","2012-12-05","2013-03-21","2012-11-16","2012-11-21","2012-11-21","2012-11-22","2012-11-29","2012-12-09","2013-01-11","2013-01-11","2013-01-14","2014-01-12","2014-01-12","2014-01-12","2014-01-13","2014-01-20","2014-01-20","2014-02-20","2014-02-20","2014-03-01","2014-04-01","2014-02-13","2014-02-13","2014-02-23","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-03-12","2014-03-12","2014-03-12","2014-03-12","2014-01-26","2014-02-22","2014-03-09","2014-07-09","2014-07-09","2013-03-22","2013-05-19","2013-05-19","2013-05-19","2013-05-19","2013-06-10","2013-08-30","2013-08-30","2013-08-30","2013-08-31","2013-09-01","2013-09-02","2013-09-02","2013-09-02","2013-09-02","2013-09-02","2013-09-02","2013-09-04","2013-09-05","2013-09-05","2013-09-15","2013-09-30","2013-10-15","2013-10-15","2013-10-23","2013-10-23","2013-10-24","2013-10-24","2013-10-24","2014-05-13","2014-05-13","2014-06-23","2014-06-23","2014-06-23","2014-07-04","2014-07-04","2014-09-12","2014-11-03","2012-10-22","2012-12-02","2012-12-25","2012-12-25","2013-01-19","2013-01-19","2013-01-26","2013-01-26","2013-01-26","2013-08-02","2013-08-02","2013-08-05","1986-01-01","1986-01-01","2013-06-14","2013-07-16","2013-10-13","2013-10-13","2013-02-07","2013-02-07","2013-03-23","2013-03-23","2013-03-23","2013-04-12","2013-04-12","2013-04-12","2013-05-07","2013-06-24","2013-06-24","2013-07-17","2014-03-18","2014-03-18","2013-03-16","2013-03-16","2013-10-21","2013-10-21","2013-12-02","2014-01-21","2014-01-21","2014-03-09","2014-01-10","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-01-19","2013-02-04","2013-02-04","2013-02-08","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-09-02","2013-09-02","2013-09-09","2013-10-12","2013-03-04","2013-03-05","2007-01-01","2013-11-16","2013-11-20","2013-11-23","2013-11-23","2013-11-23","2013-11-25","2012-09-13","2012-09-13","2012-09-13","2012-09-13","2012-10-09","2012-10-09","2013-02-28","2013-02-28","2013-04-05","2013-04-05","2013-04-05","2013-04-25","2013-05-27","2013-07-17","2013-07-17","2013-03-16","2013-03-16","2013-05-04","2013-05-04","2013-12-18","2013-10-31","2013-10-31","2013-11-22","2013-11-22","2013-04-02","2013-05-04","2013-05-04","2001-01-01","2001-01-01","2012-08-11","2012-08-11","2012-09-04","2012-09-04","2012-09-04","2013-12-28","1992-01-01","2012-09-27","2012-09-27","2013-03-01","2013-03-01","2013-03-01","2013-03-01","2013-03-13","2014-03-15","2014-03-15","2012-12-12","2012-12-12","2012-12-13","2012-12-13","2012-12-15","2014-04-24","2014-04-24","2013-01-30","2013-02-21","2013-02-25","2013-09-05","2013-09-05","2013-09-05","2013-09-14","2013-09-17","2013-09-22","2014-02-27","2014-07-05","2014-07-05","2014-07-06","2014-07-06","2014-07-06","2014-07-06","2013-10-19","2013-10-19","2013-11-01","2013-11-01","2013-11-05","2013-11-05","2013-11-05","2013-11-05","2013-09-28","2013-09-28","2013-10-11","2013-10-11","2013-10-25","2013-10-25","2013-10-26","2013-10-27","2013-10-27","2013-11-09","2013-11-09","2013-11-09","2013-11-17","2013-11-17","2013-12-15","2013-12-24","2014-01-22","2014-02-09","2013-03-30","2013-03-30","2013-05-04","2013-05-04","2013-02-21","2013-02-23","2013-03-23","2013-03-23","2013-06-07","2013-12-23","2013-12-23","2013-12-28","2013-12-28","2014-01-27","2014-01-27","2013-12-08","2013-12-08","2012-09-27","2012-09-27","2013-02-04","2012-12-30","2013-08-14","2013-09-28","2013-10-06","2013-10-17","2013-10-17","2013-10-17","2013-10-27","2013-10-27","2013-11-08","2013-11-08","2013-11-09","2013-11-09","2013-11-15","2013-11-15","2013-11-15","2013-11-15","2013-12-06","2013-12-06","2013-07-30","2013-08-04","2013-08-04","2013-08-04","2014-05-23","2014-07-04","2013-12-10","2013-12-10","2013-12-10","2014-01-18","2014-01-18","2014-02-03","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2012-08-23","2013-01-06","2014-10-31","2013-12-25","2013-11-02","2014-01-26","2013-09-17","2013-09-17","2014-04-20","2014-01-30","2014-01-30","2014-01-30","2013-10-28","2014-02-13","2014-02-05","2014-02-05","2014-02-05","2014-02-05","2013-11-14","2014-01-09","2013-04-08","2013-04-11","2013-08-25","2013-08-25","2011-12-05","2011-12-05","2012-09-19","2013-01-21","2013-06-22","2013-07-04","2012-05-01","2012-05-01","2014-01-07","2014-01-14","2014-01-14","2014-01-15","2014-01-15","2014-03-04","2014-03-04","2013-06-03","2013-06-18","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-15","2012-09-23","2012-09-23","2012-09-23","2012-09-23","2013-11-25","2013-11-25","2013-11-26","2013-11-26","2013-12-03","2013-12-03","2014-02-08","2014-02-08","2013-12-31","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2013-02-26","2013-05-23","2013-05-23","2013-06-09","2013-06-09","2013-06-23","2013-06-23","2013-01-19","2013-01-19","2013-02-04","2013-02-16","2013-03-16","2013-03-16","2014-02-25","2014-03-22","2013-12-19","2014-03-01","2014-03-09","2014-03-09","2014-03-09","2014-03-24","2014-03-24","2014-03-24","2014-03-24","2013-02-22","2013-02-22","2012-11-06","2012-11-06","2012-11-06","2012-11-22","2013-05-06","2013-05-06","2013-05-06","2013-05-06","2013-05-07","2013-05-07","2013-06-07","2013-06-07","2013-02-11","2013-02-18","2013-02-21","2013-02-21","2013-02-27","2013-03-01","2013-03-01","2013-03-01","2013-03-01","2013-03-07","2013-03-07","2013-03-19","2013-09-12","2013-09-23","2014-02-18","2014-02-18","2013-06-27","2013-06-27","2013-09-04","2013-09-04","2013-10-04","2013-10-04","2013-12-07","2013-04-02","2013-04-02","2013-05-08","2014-01-29","2014-02-15","2012-12-18","2013-01-12","2013-02-02","2013-04-10","2013-04-10","2013-04-10","2013-04-10","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-06-13","2013-08-18","2013-08-18","2013-12-01","2013-08-18","2013-08-18","2013-08-28","2013-04-10","2014-06-16","2014-06-16","2013-11-22","2013-12-10","2013-12-10","2013-12-10","2013-12-10","2013-12-10","2013-12-10","2012-08-16","2012-08-16","2012-08-21","2012-08-30","2012-12-27","2013-01-03","2013-01-03","2013-01-15","2013-01-09","2013-01-09","2013-01-17","2013-01-24","2013-01-24","2013-03-20","2013-03-20","2013-03-20","2013-03-20","2013-05-12","2013-05-12","2013-05-12","2013-05-12","2013-05-15","2013-06-12","2013-06-19","2014-04-07","2014-05-01","2014-05-08","2014-08-02","2013-11-06","2013-11-06","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2014-03-23","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-03-20","2013-03-26","2013-08-17","2013-08-17","2013-09-06","2013-09-06","2013-09-06","2013-09-06","2013-03-06","2013-03-20","2013-03-26","2013-03-26","2013-03-26","2013-03-27","2013-03-30","2013-04-01","2013-04-01","2013-05-17","2013-05-27","2013-05-28","2013-05-30","2013-05-30","2013-05-30","2013-05-31","2013-06-09","2013-06-09","2013-06-28","2013-02-05","2013-02-05","2013-03-14","2013-03-17","2013-03-17","2013-03-17","2013-04-13","2013-04-13","2013-04-13","2013-04-20","2013-04-20","2013-04-14","2013-04-19","2013-04-19","2013-04-19","2013-04-19","2013-05-19","2013-08-30","2014-04-20","2014-04-20","2014-04-20","2014-04-20","2014-04-20","2014-04-20","2014-05-04","2014-05-04","2014-05-20","2013-06-10","2013-06-10","2013-06-18","2013-06-18","2013-06-19","2013-06-19","2013-06-19","2013-06-19","2013-06-20","2013-06-26","2013-07-01","2013-07-01","2013-07-04","2013-07-28","2013-09-06","2014-05-17","2014-10-19","2013-05-01","2013-05-01","2013-05-21","2013-05-21","2013-06-07","2013-07-12","2013-08-25","2013-08-25","2011-11-01","2013-05-12","2013-03-07","2014-01-16","2014-01-16","2013-02-18","2013-03-01","2013-03-01","2013-04-16","2013-04-17","2013-04-17","2013-05-18","2013-06-29","2013-06-29","2013-08-08","2013-08-08","2013-08-08","2013-10-10","2013-10-21","2013-10-21","2014-05-09","2014-05-09","2014-05-23","2014-05-23","2013-06-22","2013-07-02","2013-07-02","2013-07-02","2013-07-02","2013-07-02","2013-07-02","2013-07-22","2013-07-22","2013-08-03","2013-08-03","2013-08-13","2013-08-13","2013-01-28","2013-01-28","2013-01-28","2013-01-28","2013-01-02","2013-01-04","2013-01-04","2013-01-21","2013-01-23","2013-01-31","2013-02-08","2013-02-09","2012-10-06","2012-12-12","2012-12-12","2012-12-12","2013-01-22","2013-02-05","1977-01-01","1977-01-01","2013-12-12","2013-12-27","2014-01-12","2013-08-02","2013-10-13","2013-10-13","2013-11-04","2013-11-13","2012-10-04","2012-10-04","2012-10-14","2012-10-19","2012-10-19","2014-04-18","2014-04-18","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-20","2014-03-10","2014-03-10","2012-12-23","2012-11-22","2013-01-13","2013-01-29","2014-02-11","2014-02-18","2014-02-23","2014-04-24","2013-07-03","2013-08-21","2013-08-21","2013-09-29","2013-09-29","2013-10-09","2013-10-09","2013-10-11","2013-10-11","2014-02-23","2014-02-23","2012-11-21","2012-11-21","2012-11-21","2012-11-22","2012-11-22","2013-01-13","2013-02-07","2013-02-07","2013-02-24","2013-02-24","2013-02-24","2013-03-02","2013-01-31","2013-02-10","2013-02-10","2013-02-13","2013-05-20","2013-05-20","2013-02-26","2013-05-14","2013-05-14","2013-06-18","2013-06-18","2013-06-18","2013-06-18","2012-12-30","2013-02-14","2012-12-22","2012-12-22","2013-01-24","2013-12-25","2013-12-25","2014-03-26","2014-03-26","2012-11-12","2012-11-12","2012-11-26","2012-12-18","2012-12-27","2012-12-27","2013-02-18","2013-02-18","2013-01-31","2013-01-31","2013-02-24","2013-02-24","2013-04-16","2013-04-16","2013-04-16","2013-04-16","2013-04-16","2013-04-27","2013-04-29","2012-07-29","2012-08-04","2012-08-04","2012-08-12","2007-10-01","2013-04-11","2013-04-28","2013-04-28","2013-04-28","2013-04-29","2013-04-29","2013-05-18","2013-05-27","2013-05-27","2013-05-28","2013-05-28","2013-05-30","2013-06-01","2012-07-23","2012-07-23","2013-02-05","2013-09-26","2013-09-26","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2014-03-08","2014-07-14","2013-06-21","2013-06-21","2013-06-30","2013-07-01","2013-08-19","2013-08-22","2013-08-28","2013-08-28","2013-09-05","2013-09-05","2013-09-12","2013-10-10","2013-10-10","2013-10-10","2013-10-10","2013-11-23","2013-11-23","2013-11-23","2013-11-23","2013-05-30","2013-05-30","2013-06-02","2013-10-31","2012-10-30","2012-10-30","2014-02-02","2014-04-18","2014-06-01","2014-06-01","2014-07-10","2014-07-10","2014-03-19","2014-03-19","2014-03-27","2014-03-27","2014-05-26","2014-05-26","2014-05-26","2014-06-04","2014-06-04","2014-06-04","2014-07-18","2013-08-12","2013-08-12","2013-09-20","2013-09-20","2013-09-20","2013-09-20","2013-05-12","2013-05-12","2013-05-12","2013-05-12","2013-05-13","2013-05-13","2013-05-14","2013-05-17","2013-07-25","2013-12-13","2013-12-14","2014-01-11","2014-01-22","2014-01-22","2014-02-24","2014-02-24","2014-03-08","2014-05-27","2013-03-30","2013-03-30","2013-04-04","2013-04-04","2013-05-29","2013-05-29","2013-06-16","2013-06-16","2013-08-13","2013-05-07","2013-05-07","2013-05-13","2013-01-10","2013-03-18","2014-03-14","2014-03-14","2014-04-21","2014-04-22","2013-07-08","2013-07-08","2013-07-08","2013-07-08","2013-07-13","2013-07-13","2012-08-20","2012-08-20","2012-12-16","2014-04-17","2014-04-17","2014-04-18","2014-04-18","2014-07-16","2014-07-31","2014-08-15","2013-05-21","2013-05-21","2013-10-15","2013-05-09","2013-07-01","2013-07-04","2013-07-04","2013-07-28","2013-07-28","2013-07-31","2013-07-13","2013-07-13","2013-07-31","2013-01-19","2013-01-19","2013-01-19","2014-07-09","2014-07-09","2014-07-09","2014-08-17","2014-08-17","2013-04-12","2013-05-01","2013-02-19","2013-02-19","2013-03-15","2013-03-15","2013-10-17","2013-10-17","2013-10-25","2013-10-25","2013-05-09","2013-05-09","2012-11-17","2012-11-17","2012-12-03","2013-03-10","2013-03-10","2014-08-24","2012-11-24","2012-11-24","2013-01-05","2013-02-07","2013-02-20","2013-03-06","2013-04-22","2013-04-22","2013-04-22","2013-04-22","2013-06-08","2013-06-08","2013-09-20","2014-05-28","2014-05-28","2014-07-22","2014-08-02","2013-08-18","2013-09-02","2012-12-14","2012-12-15","2013-05-05","2013-05-05","2013-05-05","2013-05-07","2013-07-01","2013-07-01","2013-07-01","2013-07-01","2013-09-26","2013-09-26","2014-05-11","2014-01-01","2014-01-14","2014-01-14","2014-03-14","2014-04-05","2013-05-01","2014-01-13","2014-01-13","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-03-09","2014-03-09","2014-04-08","2014-04-21","2014-05-21","2014-06-03","2014-06-03","2014-01-29","2014-01-29","2014-02-15","2014-02-15","2014-02-20","2014-02-22","2014-03-04","2014-04-18","2013-01-26","2013-01-26","2013-02-03","2013-02-03","2013-04-17","2013-04-25","2013-12-19","2014-01-15","2014-02-01","2014-02-01","2014-02-01","2014-02-28","2014-02-28","1994-04-01","2013-05-23","2013-05-23","2013-05-29","2013-06-11","2013-06-11","2013-07-07","2013-07-22","2013-08-10","2013-09-15","2013-09-15","2013-09-19","2013-10-06","2013-10-06","2013-10-07","2013-10-07","2014-01-13","2014-01-28","2013-07-16","2013-07-18","2013-07-30","2013-07-30","2012-10-05","2012-10-05","2012-10-05","2013-03-06","2013-04-13","2013-04-13","2013-04-14","2013-04-14","2013-07-06","2013-06-03","2013-06-03","2013-11-16","2013-02-16","2013-02-16","2013-03-06","2013-03-06","2013-03-15","2013-05-10","2013-05-10","2013-06-09","2013-06-16","2013-07-31","2013-10-07","2013-10-11","2013-10-11","2013-10-11","2013-10-12","2013-10-16","2013-10-09","2013-10-09","2013-10-09","2013-10-09","2013-10-15","2013-10-15","2013-10-15","2013-11-15","2013-12-29","2013-12-29","2013-12-29","2013-07-13","2013-07-25","2013-07-25","2013-08-13","2013-08-13","2013-09-03","2013-09-03","2013-10-31","2013-10-31","2013-03-23","2013-03-23","2013-04-02","2013-04-02","1982-01-01","2013-03-08","2013-03-08","2013-04-17","2013-04-27","2013-05-12","2013-05-16","2013-05-20","2013-05-25","2013-06-23","2013-06-23","2013-08-15","2013-08-15","2013-05-09","2013-05-09","2013-05-24","2013-05-24","2013-06-02","2013-06-02","2012-12-18","2012-12-26","2013-01-01","2013-01-01","2013-01-03","2013-01-05","2013-01-16","2013-01-21","2013-01-27","2013-01-27","2013-01-27","2013-01-29","2013-02-04","2013-02-04","2013-02-04","2013-02-04"],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D",null,null,null,null,null,null,null,null,null,null,null,"M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D","D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D","D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"M","M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"M","M",null,null,null,null,null,null,"M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D","D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"M","M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D","D","D","D",null,null,null,null,null,null,null,null,"D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"D",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"M",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[2,2,8,3,3,3,22,3,21,58,125,1,1,23,27,3,51,51,52,52,80,90,109,109,109,-61,-5,-5,-5,-5,1,7,7,3,3,3,3,27,27,46,51,51,-4088,23,23,23,84,84,97,97,97,61,61,61,61,-4,-4,13,15,22,22,22,22,34,38,38,-569,-25,3,3,51,51,115,117,134,174,21,21,-4060,1,1,4,4,23,36,36,36,5,2,2,2,2,18,18,35,35,-782,-782,13,17,17,17,17,48,48,48,48,137,137,137,137,48,2,7,7,8,15,25,58,58,61,2,2,2,3,10,10,41,41,50,81,7,7,17,18,18,18,18,18,18,18,18,34,34,34,34,2,29,44,166,166,-2,57,57,57,57,79,2,2,2,3,4,5,5,5,5,5,5,7,8,8,18,33,48,48,56,56,57,57,57,-9,-9,33,33,33,44,44,114,166,21,62,85,85,15,15,22,22,22,3,3,6,-10011,-10011,16,48,137,137,4,4,48,48,48,68,68,68,93,141,141,164,7,7,140,140,29,29,71,121,121,168,77,15,15,15,15,15,31,31,35,-19,-19,-19,-19,-19,-19,39,39,46,79,3,4,-2489,23,27,30,30,30,32,12,12,12,12,38,38,-13,-13,24,24,24,44,76,127,127,25,25,74,74,-2,15,15,37,37,18,50,50,-4218,-4218,23,23,47,47,47,38,-7560,16,16,171,171,171,171,183,-2,-2,1,1,2,2,4,44,44,18,40,44,10,10,10,19,22,27,7,135,135,136,136,136,136,14,14,27,27,31,31,31,31,2,2,15,15,29,29,3,4,4,17,17,17,25,25,53,62,1,19,16,16,51,51,30,32,60,60,136,22,22,27,27,57,57,39,39,9,9,139,42,-11,35,-7,5,5,5,15,15,27,27,28,28,34,34,34,34,55,55,23,28,28,28,31,73,2,2,2,41,41,57,16,16,16,16,-106,51,174,29,90,19,2,2,94,64,64,64,15,123,52,52,52,52,13,69,30,33,169,169,-277,-277,13,137,-1,12,-609,-609,8,15,15,16,16,64,64,21,36,1,1,1,1,1,1,9,9,9,9,29,29,30,30,37,37,104,104,12,13,13,13,13,18,15,15,32,32,46,46,90,90,106,118,146,146,18,43,-10,63,71,71,71,86,86,86,86,14,14,14,14,14,30,-1,-1,-1,-1,1,1,32,32,6,13,16,16,22,24,24,24,24,30,30,42,3,14,162,162,14,14,83,83,113,113,177,68,68,104,160,177,12,37,58,125,125,125,125,55,55,55,55,55,55,55,55,14,14,119,46,46,56,-2,173,173,14,32,32,32,32,32,32,17,17,22,31,27,34,34,46,16,16,24,31,31,86,86,86,86,139,139,139,139,142,170,177,80,104,27,113,13,13,19,19,19,19,19,150,28,28,28,28,65,71,16,16,36,36,36,36,3,17,23,23,23,24,27,29,29,75,85,14,16,16,16,17,26,26,45,11,11,48,51,51,51,78,78,78,85,85,22,27,27,27,27,57,43,77,77,77,77,77,77,91,91,107,-9,-9,-1,-1,1,1,1,1,2,8,13,13,16,40,80,28,183,27,27,47,47,64,99,11,11,-418,141,5,3,3,9,20,20,66,67,67,98,140,140,180,180,180,14,25,25,71,71,85,85,20,30,30,30,30,30,30,50,50,62,62,72,72,34,34,34,34,2,4,4,21,23,31,39,40,29,96,96,96,137,151,-13469,-13469,26,41,57,12,3,3,25,34,3,3,13,18,18,21,21,110,110,110,110,111,115,115,44,58,110,126,46,53,58,118,21,70,70,109,109,119,119,121,121,12,12,56,56,56,57,57,21,46,46,63,63,63,69,48,58,58,61,27,27,-1,77,77,112,112,112,112,50,96,97,97,130,64,64,155,155,15,15,29,51,60,60,113,113,27,27,51,51,14,14,14,14,14,25,27,1,7,7,15,-2011,9,26,26,26,27,27,46,55,55,56,56,58,60,-188,-188,10,12,12,27,27,27,27,175,107,22,22,31,32,11,14,20,20,28,28,35,63,63,63,63,107,107,107,107,4,4,7,158,42,42,7,82,126,126,165,165,60,60,68,68,40,40,40,49,49,49,93,5,5,44,44,44,44,17,17,17,17,18,18,19,22,91,10,11,39,50,50,83,83,95,175,33,33,38,38,93,93,111,111,-2,71,71,77,54,30,32,32,70,71,3,3,3,3,8,8,43,43,161,16,16,17,17,106,121,136,25,25,172,1,54,57,57,81,81,84,36,36,54,32,32,32,112,112,112,151,151,60,79,19,19,43,43,16,16,24,24,35,35,41,41,57,154,154,-9,47,47,89,122,1,15,15,15,15,15,62,62,24,15,15,70,81,46,61,1,2,1,1,1,3,58,58,58,58,145,145,110,17,30,30,89,111,-258,-1,-1,2,2,2,2,2,2,2,2,2,55,55,85,98,128,141,141,3,3,20,20,25,27,37,82,5,5,13,13,86,94,1,28,22,22,22,49,49,-6970,23,23,29,42,42,68,83,102,15,15,19,36,36,37,37,135,150,10,12,24,24,17,17,17,18,56,56,57,57,140,16,16,182,29,29,47,47,56,112,112,142,149,194,22,26,26,26,27,31,19,19,19,19,25,25,25,56,100,100,100,4,16,16,35,35,56,56,114,114,51,51,61,61,-11381,9,9,49,59,74,78,82,87,116,116,169,169,14,14,29,29,38,38,2,10,16,16,18,20,31,36,42,42,42,44,50,50,50,50],[null,null,"2014-01-11","2012-08-30",null,"2012-08-30",null,null,null,null,null,"2013-02-12","2013-02-12",null,null,null,"2014-02-22","2014-02-20",null,null,"2014-03-21","2014-03-31","2014-04-22","2014-04-22","2014-04-20",null,"2012-09-07","2012-09-07","2012-09-07","2012-09-07",null,null,null,null,null,null,null,"2012-12-26","2012-12-26",null,null,null,null,"2014-03-09","2014-03-16","2014-03-16","2014-05-09","2014-05-09",null,null,null,null,null,null,null,"2013-06-02","2013-06-02","2013-06-02",null,null,null,null,null,null,null,null,null,null,null,null,"2013-11-10","2013-11-10","2013-12-17","2013-12-17","2014-01-03",null,"2013-11-06","2013-11-06",null,"2013-02-12","2013-02-12","2013-03-19","2013-03-19","2013-03-07",null,null,null,"2013-12-09",null,null,null,null,null,null,null,null,null,null,"2012-08-03","2012-09-14","2012-09-14","2012-09-14","2012-09-14","2012-09-30","2012-09-30","2012-10-06","2012-10-06",null,"2012-12-16","2012-12-16",null,"2013-03-21",null,"2012-11-21",null,null,null,null,null,null,"2013-01-14",null,null,null,null,"2014-02-08","2014-02-08","2014-03-12","2014-03-12",null,null,"2014-06-03","2014-06-03","2014-02-23","2014-06-12","2014-06-12","2014-06-03","2014-06-03","2014-06-12","2014-06-12","2014-06-03","2014-06-03","2014-03-27","2014-03-27","2014-03-27","2014-03-27",null,null,null,"2014-07-12","2014-07-12","2013-03-23",null,null,null,null,null,null,null,null,null,null,"2013-09-03",null,null,null,null,null,null,"2013-10-08","2013-10-08",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-01-16","2013-01-16",null,null,null,"2013-01-30","2013-01-30",null,null,null,null,null,"2013-06-15",null,"2013-10-15","2013-10-15","2013-02-07","2013-02-07","2013-03-26",null,null,null,null,null,"2013-05-24","2013-06-24","2013-06-24","2013-07-26","2014-03-18","2014-03-18","2013-03-21","2013-03-21","2013-11-04","2013-11-04",null,"2014-02-10","2014-02-10","2014-03-09",null,null,null,null,null,null,null,null,"2013-02-08","2013-07-26","2013-07-26","2013-07-24","2013-07-24","2013-07-26","2013-07-26","2013-09-06","2013-09-06","2013-09-24","2013-10-31","2013-03-05","2013-03-06",null,"2013-11-16",null,null,null,null,null,"2013-01-02","2013-01-02","2013-01-02","2013-01-02","2012-10-15","2012-10-15","2013-09-14","2013-09-14","2013-04-16","2013-04-16","2013-04-16","2013-04-25","2013-06-03","2013-09-14","2013-09-14",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2012-08-19","2012-08-19","2013-01-25","2013-01-25","2013-01-25",null,null,"2012-10-06","2012-10-06",null,null,null,null,null,"2014-03-30","2014-03-30","2012-12-19","2012-12-19","2012-12-14","2012-12-14","2012-12-16",null,null,null,"2013-02-21",null,null,"2013-09-27","2013-09-27",null,null,"2013-09-23","2014-03-06","2014-07-06","2014-07-06",null,null,null,null,null,null,null,"2013-11-08","2013-11-22","2013-11-22","2013-11-22","2013-11-22","2013-09-30","2013-09-30","2013-10-25","2013-10-25","2013-11-29","2013-11-29",null,null,null,null,null,null,"2013-11-17","2013-11-17","2013-12-15",null,null,null,"2013-04-13","2013-04-13",null,null,"2013-02-23",null,null,null,"2013-06-13",null,null,"2014-02-03","2014-02-03",null,null,null,null,"2012-10-07","2012-10-07","2013-02-13",null,null,null,null,"2013-12-23","2013-12-23","2013-12-23","2013-11-08","2013-11-08","2013-12-06","2013-12-06","2013-12-04","2013-12-04","2013-12-14","2013-12-14","2013-12-23","2013-12-23","2013-12-23","2013-12-23","2013-08-03",null,null,null,null,null,"2014-02-24","2014-02-24","2014-02-24","2014-02-24","2014-02-24",null,"2014-04-04","2014-04-04","2014-04-04","2014-04-04",null,null,"2014-10-31",null,null,null,"2013-09-29","2013-09-29",null,null,null,null,null,null,"2014-03-03","2014-03-03","2014-03-03","2014-03-03",null,null,"2013-04-08","2013-04-11",null,null,"2013-02-20","2013-02-20","2012-09-24","2013-01-24",null,"2013-07-06",null,null,"2014-01-08","2014-01-28","2014-01-28","2014-07-29","2014-07-29","2014-07-29","2014-07-29","2013-06-03","2013-06-18","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2012-09-29","2013-12-23","2013-12-23","2013-12-09","2013-12-09","2013-12-10","2013-12-10","2014-02-22","2014-02-22","2013-12-31","2014-01-01","2014-01-01","2014-01-01","2014-01-01","2013-03-09","2013-05-23","2013-05-23","2013-06-09","2013-06-09","2013-06-23","2013-06-23","2013-01-19","2013-01-19","2013-02-08",null,"2013-03-16","2013-03-16",null,"2014-03-22",null,"2014-03-08","2014-04-06","2014-04-06",null,"2014-03-24","2014-03-26","2014-03-28","2014-03-26","2013-02-22","2013-02-22","2012-12-13","2012-12-13",null,null,"2013-05-16","2013-05-16","2013-05-16","2013-05-16","2013-05-10","2013-05-10","2013-06-07","2013-06-07","2013-02-11","2013-02-18","2013-03-20","2013-03-20","2013-02-27","2013-03-07","2013-03-07","2013-03-07","2013-03-07","2013-03-07","2013-03-07","2013-03-19",null,null,"2014-02-18","2014-02-18","2013-06-27","2013-06-27","2013-09-18","2013-09-18","2013-10-30","2013-10-30","2013-12-07",null,null,null,null,null,"2012-12-18","2013-01-12","2013-02-02","2013-06-05","2013-06-05","2013-06-05","2013-06-05","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-06-25","2013-08-31","2013-08-31",null,"2013-09-10","2013-09-10","2013-09-10",null,null,null,"2013-11-22","2013-12-21","2013-12-21","2013-12-21","2013-12-21","2013-12-21","2013-12-21","2012-08-19","2012-08-19",null,"2012-08-30",null,"2013-01-11","2013-01-11",null,"2013-02-19","2013-02-19",null,"2013-02-27","2013-02-27","2013-04-16","2013-04-16","2013-04-16","2013-04-16","2013-05-19","2013-05-19","2013-05-19","2013-05-19",null,null,null,null,null,null,null,"2013-11-17","2013-11-17",null,null,null,null,null,"2014-03-23",null,null,null,null,"2013-03-26","2013-03-27","2013-09-06","2013-09-06","2013-10-11","2013-10-11","2013-10-11","2013-10-11","2013-03-06",null,"2013-06-18","2013-06-18","2013-06-18","2013-03-27",null,"2013-04-11","2013-04-11","2013-05-18","2013-06-18","2013-05-28",null,null,null,"2013-05-31",null,null,"2013-06-28","2013-02-13","2013-02-13",null,"2013-03-30","2013-03-30","2013-03-17","2013-04-14","2013-04-14","2013-04-14","2013-06-13","2013-06-13",null,"2013-05-31","2013-05-31","2013-05-31","2013-05-31",null,null,"2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-06-23","2014-05-23","2013-07-01","2013-07-01","2013-07-16","2013-07-16","2013-06-20","2013-06-20","2013-06-19","2013-06-19",null,"2013-06-26","2013-07-01","2013-07-01","2013-07-04",null,null,null,null,null,null,"2013-05-21","2013-05-21","2013-06-17",null,"2013-08-25","2013-08-25",null,null,"2013-03-07","2014-01-20","2014-01-20","2013-02-19",null,null,"2013-04-16","2013-04-17","2013-04-19",null,"2013-07-04","2013-07-04",null,null,"2013-08-08","2013-10-10","2013-10-31","2013-10-31","2014-06-01","2014-06-01","2014-06-20","2014-06-20","2013-06-22","2013-07-09","2013-07-09","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-02-11","2013-02-11","2013-02-11","2013-02-11","2013-01-02","2013-01-15","2013-01-15","2013-01-21","2013-01-23","2013-01-31","2013-02-08","2013-02-09",null,null,null,null,"2013-01-25","2013-02-10",null,null,"2013-12-12",null,null,"2013-08-02",null,null,null,null,"2012-10-06","2012-10-06","2012-10-14","2012-10-19","2012-10-19",null,null,"2013-03-20","2013-03-19","2013-03-19","2013-03-19","2013-03-20",null,null,null,"2012-12-14",null,null,"2014-02-11","2014-02-18",null,null,"2013-07-05",null,null,"2013-09-29","2013-09-29","2013-10-09","2013-10-09","2013-10-11","2013-10-11",null,null,"2012-11-26","2012-11-21","2012-11-21",null,"2012-11-26","2013-01-19","2013-03-03","2013-03-03","2013-03-03","2013-03-03","2013-02-26","2013-03-02",null,null,null,null,"2013-05-24","2013-05-24","2013-02-26","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07",null,null,null,null,null,"2014-04-24","2014-04-24",null,null,"2012-11-13","2012-11-13",null,null,null,null,null,null,"2013-02-09","2013-02-09","2013-05-31","2013-05-31",null,null,"2013-05-02","2013-05-02","2013-05-02",null,"2013-05-01",null,"2012-08-12","2012-08-12",null,null,null,"2013-05-11","2013-05-11","2013-05-11","2013-05-15","2013-05-15",null,"2013-05-31","2013-05-31","2013-06-01","2013-06-01",null,null,null,null,null,null,null,"2013-10-11","2013-10-11","2013-10-11","2013-10-11","2014-03-08",null,"2013-07-15","2013-07-15",null,"2013-07-15","2013-08-23","2013-08-22","2013-09-08","2013-09-08",null,null,null,"2013-10-13",null,"2014-01-18","2014-01-18","2013-12-02","2013-12-02","2013-12-02","2013-12-02",null,null,null,null,"2012-10-31","2012-10-31",null,null,null,null,null,null,"2014-05-23","2014-05-23","2014-05-23","2014-05-23","2014-06-04","2014-06-04",null,null,"2014-06-17","2014-06-17","2014-07-24","2013-08-22","2013-08-22",null,null,null,null,"2013-07-17","2013-07-17","2013-07-17","2013-07-17","2013-07-17","2013-07-17",null,null,"2013-08-06","2013-12-13","2013-12-15",null,"2014-01-25","2014-01-25","2014-04-08","2014-04-08",null,null,"2013-04-06","2013-04-06","2013-04-15","2013-04-15",null,null,null,null,null,"2013-05-21","2013-05-21",null,"2013-01-10","2013-03-18","2014-03-14","2014-03-14",null,null,"2013-07-08","2013-07-08","2013-07-08","2013-07-08","2013-07-13","2013-07-13","2012-09-27","2012-09-27",null,null,null,"2014-07-31","2014-07-31","2014-07-25",null,"2014-08-27","2013-05-26","2013-05-26",null,null,null,"2013-07-28","2013-07-28","2013-07-31","2013-07-31",null,null,null,"2013-07-31",null,null,null,"2014-07-16",null,null,"2014-08-24","2014-08-24",null,null,"2013-03-22","2013-03-22",null,null,"2013-11-04","2013-11-04",null,null,null,null,null,null,null,"2013-04-10","2013-04-10",null,null,null,null,null,null,null,"2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06","2013-07-06",null,"2014-09-22","2014-09-22","2014-08-02",null,null,null,null,null,null,"2013-05-07","2013-05-07",null,"2013-09-26","2013-09-26","2013-10-04","2013-10-04","2013-11-11","2013-11-11",null,null,null,null,null,"2014-04-09",null,"2014-02-10","2014-02-10",null,"2014-02-18","2014-02-18",null,null,"2014-02-08","2014-02-08","2014-02-09","2014-02-09","2014-03-15","2014-03-15",null,null,null,"2014-06-10","2014-06-10","2014-07-16","2014-07-16","2014-02-17",null,null,null,null,null,null,null,null,null,"2013-04-19","2013-04-26",null,null,"2014-02-18","2014-02-18","2014-02-06","2014-03-01","2014-03-01",null,null,null,null,"2013-10-15","2013-10-15","2013-07-08","2013-07-22","2013-08-10","2013-09-16","2013-09-16",null,"2014-03-03","2014-03-03","2013-10-11","2013-10-11","2014-01-14",null,"2013-07-20",null,null,null,null,null,null,"2013-03-06","2013-05-12","2013-05-12","2013-07-06","2013-07-06",null,"2013-07-16","2013-07-16","2013-11-16","2013-03-17","2013-03-17","2013-03-26","2013-03-26","2013-03-15","2013-07-07","2013-07-07",null,null,null,"2013-10-07","2013-10-12","2013-10-12","2013-10-12","2013-10-13",null,"2013-11-15","2013-11-15","2013-11-15","2013-10-09","2013-11-02","2013-11-02","2013-11-02","2013-11-15","2014-02-19","2014-02-19","2014-02-19",null,"2013-07-25","2013-07-25",null,null,"2013-09-19","2013-09-19",null,null,"2013-03-25","2013-03-25","2013-04-21","2013-04-21",null,"2013-03-22","2013-03-22",null,"2013-04-27","2013-05-12","2013-05-16","2013-05-20",null,"2013-08-13","2013-08-13","2013-08-29","2013-08-29","2013-05-11","2013-05-11",null,null,"2013-06-03","2013-06-05","2012-12-18","2012-12-26","2013-01-19","2013-01-19","2013-01-09","2013-01-05","2013-01-16","2013-01-24","2013-01-27","2013-01-28","2013-01-28",null,"2013-02-25","2013-02-25","2013-02-25","2013-02-25"],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,10,26,null,26,null,null,null,null,null,1,1,null,null,null,53,51,null,null,80,90,112,112,110,null,1,1,1,1,null,null,null,null,null,null,null,27,27,null,null,null,null,23,30,30,84,84,null,null,null,null,null,null,null,14,14,14,null,null,null,null,null,null,null,null,null,null,null,null,80,80,117,117,134,null,45,45,null,1,1,36,36,24,null,null,null,5,null,null,null,null,null,null,null,null,null,null,13,55,55,55,55,71,71,77,77,null,148,148,null,48,null,7,null,null,null,null,null,null,61,null,null,null,null,29,29,61,61,null,null,117,117,17,126,126,117,117,126,126,117,117,49,49,49,49,null,null,null,169,169,-1,null,null,null,null,null,null,null,null,null,null,6,null,null,null,null,null,null,41,41,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,107,107,null,null,null,26,26,null,null,null,null,null,17,null,139,139,4,4,51,null,null,null,null,null,110,141,141,173,7,7,145,145,43,43,null,141,141,168,null,null,null,null,null,null,null,null,35,1,1,-2,-2,1,1,43,43,61,98,4,5,null,23,null,null,null,null,null,123,123,123,123,44,44,186,186,35,35,35,44,83,186,186,null,null,null,null,null,null,null,null,null,null,null,null,null,null,31,31,190,190,190,null,null,25,25,null,null,null,null,null,14,14,8,8,3,3,5,null,null,null,40,null,null,32,32,null,null,28,14,136,136,null,null,null,null,null,null,null,34,48,48,48,48,4,4,29,29,64,64,null,null,null,null,null,null,25,25,53,null,null,null,30,30,null,null,32,null,null,null,142,null,null,64,64,null,null,null,null,19,19,148,null,null,null,null,72,72,72,27,27,55,55,53,53,63,63,72,72,72,72,27,null,null,null,null,null,78,78,78,78,78,null,175,175,175,175,null,null,174,null,null,null,14,14,null,null,null,null,null,null,78,78,78,78,null,null,30,33,null,null,167,167,18,140,null,14,null,null,9,29,29,211,211,211,211,21,36,15,15,15,15,15,15,15,15,15,15,57,57,43,43,44,44,118,118,12,13,13,13,13,29,15,15,32,32,46,46,90,90,110,null,146,146,null,43,null,70,99,99,null,86,88,90,88,14,14,51,51,null,null,10,10,10,10,4,4,32,32,6,13,43,43,22,30,30,30,30,30,30,42,null,null,162,162,14,14,97,97,139,139,177,null,null,null,null,null,12,37,58,181,181,181,181,67,67,67,67,67,67,67,67,27,27,null,69,69,69,null,null,null,14,43,43,43,43,43,43,20,20,null,31,null,42,42,null,57,57,null,65,65,113,113,113,113,146,146,146,146,null,null,null,null,null,null,null,24,24,null,null,null,null,null,150,null,null,null,null,71,72,36,36,71,71,71,71,3,null,107,107,107,24,null,39,39,76,107,14,null,null,null,17,null,null,45,19,19,null,64,64,51,79,79,79,139,139,null,69,69,69,69,null,null,141,141,141,141,141,141,141,141,110,13,13,28,28,2,2,1,1,null,8,13,13,16,null,null,null,null,null,null,47,47,74,null,11,11,null,null,5,7,7,10,null,null,66,67,69,null,145,145,null,null,180,14,35,35,94,94,113,113,20,37,37,76,76,76,76,76,76,76,76,76,76,48,48,48,48,2,15,15,21,23,31,39,40,null,null,null,null,140,156,null,null,26,null,null,12,null,null,null,null,5,5,13,18,18,null,null,111,110,110,110,111,null,null,null,80,null,null,46,53,null,null,23,null,null,109,109,119,119,121,121,null,null,61,56,56,null,61,27,70,70,70,70,65,69,null,null,null,null,31,31,-1,131,131,131,131,131,131,null,null,null,null,null,184,184,null,null,16,16,null,null,null,null,null,null,36,36,147,147,null,null,30,30,30,null,29,null,15,15,null,null,null,39,39,39,43,43,null,59,59,60,60,null,null,null,null,null,null,null,27,27,27,27,175,null,46,46,null,46,15,14,31,31,null,null,null,66,null,163,163,116,116,116,116,null,null,null,null,43,43,null,null,null,null,null,null,125,125,125,125,49,49,null,null,62,62,99,15,15,null,null,null,null,83,83,83,83,83,83,null,null,103,10,12,null,53,53,126,126,null,null,40,40,49,49,null,null,null,null,null,85,85,null,54,30,32,32,null,null,3,3,3,3,8,8,81,81,null,null,null,121,121,115,null,148,30,30,null,null,null,81,81,84,84,null,null,null,54,null,null,null,119,null,null,158,158,null,null,50,50,null,null,34,34,null,null,null,null,null,null,null,185,185,null,null,null,null,null,null,null,90,90,90,90,90,90,null,132,132,81,null,null,null,null,null,null,3,3,null,145,145,153,153,191,191,null,null,null,null,null,115,null,28,28,null,36,36,null,null,26,26,27,27,61,61,null,null,null,148,148,171,171,22,null,null,null,null,null,null,null,null,null,88,95,null,null,39,39,27,50,50,null,null,null,null,168,168,69,83,102,16,16,null,184,184,41,41,136,null,14,null,null,null,null,null,null,18,85,85,140,140,null,59,59,182,58,58,67,67,56,170,170,null,null,null,22,27,27,27,28,null,56,56,56,19,43,43,43,56,152,152,152,null,16,16,null,null,72,72,null,null,53,53,80,80,null,23,23,null,59,74,78,82,null,167,167,183,183,16,16,null,null,39,41,2,10,34,34,24,20,31,39,42,43,43,null,71,71,71,71],[null,null,3,24,null,24,null,null,null,null,null,1,1,null,null,null,3,1,null,null,1,1,4,4,2,null,6,6,6,6,null,null,null,null,null,null,null,1,1,null,null,null,null,1,8,8,1,1,null,null,null,null,null,null,null,18,18,2,null,null,null,null,null,null,null,null,null,null,null,null,30,30,3,1,1,null,25,25,null,1,1,33,33,2,null,null,null,1,null,null,null,null,null,null,null,null,null,null,1,39,39,39,39,24,24,30,30,null,12,12,null,1,null,1,null,null,null,null,null,null,1,null,null,null,null,20,20,21,21,null,null,111,111,1,109,109,100,100,109,109,100,100,16,16,16,16,null,null,null,4,4,2,null,null,null,null,null,null,null,null,null,null,2,null,null,null,null,null,null,34,34,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,23,23,null,null,null,5,5,null,null,null,null,null,2,null,3,3,1,1,4,null,null,null,null,null,18,1,1,10,1,1,6,6,15,15,null,21,21,1,null,null,null,null,null,null,null,null,1,20,20,18,18,20,20,5,5,16,20,2,2,null,1,null,null,null,null,null,112,112,112,112,7,7,199,199,12,12,12,1,8,60,60,null,null,null,null,null,null,null,null,null,null,null,null,null,null,9,9,144,144,144,null,null,10,10,null,null,null,null,null,16,16,8,8,2,2,2,null,null,null,1,null,null,23,23,null,null,2,8,2,2,null,null,null,null,null,null,null,8,18,18,18,18,3,3,15,15,36,36,null,null,null,null,null,null,1,1,1,null,null,null,15,15,null,null,3,null,null,null,7,null,null,38,38,null,null,null,null,11,11,10,null,null,null,null,68,68,68,13,13,29,29,26,26,30,30,39,39,18,18,5,null,null,null,null,null,77,77,77,38,38,null,160,160,160,160,null,null,1,null,null,null,13,13,null,null,null,null,null,null,27,27,27,27,null,null,1,1,null,null,444,444,6,4,null,3,null,null,2,15,15,196,196,148,148,1,1,15,15,15,15,15,15,7,7,7,7,29,29,14,14,8,8,15,15,1,1,1,1,1,12,1,1,1,1,1,1,1,1,5,null,1,1,null,1,null,8,29,29,null,1,3,5,3,1,1,38,38,null,null,11,11,11,11,4,4,1,1,1,1,28,28,1,7,7,7,7,1,1,1,null,null,1,1,1,1,15,15,27,27,1,null,null,null,null,null,1,1,1,57,57,57,57,13,13,13,13,13,13,13,13,14,14,null,24,24,14,null,null,null,1,12,12,12,12,12,12,4,4,null,1,null,9,9,null,42,42,null,35,35,28,28,28,28,8,8,8,8,null,null,null,null,null,null,null,12,12,null,null,null,null,null,1,null,null,null,null,7,2,21,21,36,36,36,36,1,null,85,85,85,1,null,11,11,2,23,1,null,null,null,1,null,null,1,9,9,null,14,14,1,2,2,2,55,55,null,43,43,43,43,null,null,65,65,65,65,65,65,51,51,4,22,22,29,29,2,2,1,1,null,1,1,1,1,null,null,null,null,null,null,1,1,11,null,1,1,null,null,1,5,5,2,null,null,1,1,3,null,6,6,null,null,1,1,11,11,24,24,29,29,1,8,8,47,47,47,47,27,27,15,15,5,5,15,15,15,15,1,12,12,1,1,1,1,1,null,null,null,null,4,6,null,null,1,null,null,1,null,null,null,null,3,3,1,1,1,null,null,2,1,1,1,1,null,null,null,23,null,null,1,1,null,null,3,null,null,1,1,1,1,1,1,null,null,6,1,1,null,5,7,25,25,8,8,3,1,null,null,null,null,5,5,1,55,55,20,20,20,20,null,null,null,null,null,121,121,null,null,2,2,null,null,null,null,null,null,10,10,97,97,null,null,17,17,17,null,3,null,9,9,null,null,null,14,14,14,17,17,null,5,5,5,5,null,null,null,null,null,null,null,1,1,1,1,1,null,25,25,null,15,5,1,12,12,null,null,null,4,null,101,101,10,10,10,10,null,null,null,null,2,2,null,null,null,null,null,null,66,66,58,58,10,10,null,null,14,14,7,11,11,null,null,null,null,67,67,67,67,66,66,null,null,13,1,2,null,4,4,44,44,null,null,8,8,12,12,null,null,null,null,null,15,15,null,1,1,1,1,null,null,1,1,1,1,1,1,39,39,null,null,null,105,105,10,null,13,6,6,null,null,null,25,25,4,4,null,null,null,1,null,null,null,8,null,null,8,8,null,null,32,32,null,null,19,19,null,null,null,null,null,null,null,32,32,null,null,null,null,null,null,null,76,76,76,76,29,29,null,118,118,12,null,null,null,null,null,null,3,3,null,88,88,96,96,47,47,null,null,null,null,null,5,null,29,29,null,35,35,null,null,25,25,26,26,7,7,null,null,null,8,8,169,169,3,null,null,null,null,null,null,null,null,null,3,2,null,null,18,18,6,2,2,null,null,null,null,127,127,2,1,1,2,2,null,149,149,5,5,2,null,5,null,null,null,null,null,null,1,30,30,84,84,null,44,44,1,30,30,21,21,1,59,59,null,null,null,1,2,2,2,2,null,38,38,38,1,19,19,19,1,53,53,53,null,1,1,null,null,17,17,null,null,3,3,20,20,null,15,15,null,1,1,1,1,null,52,52,15,15,3,3,null,null,2,4,1,1,19,19,7,1,1,4,1,2,2,null,22,22,22,22],[null,null,"DAYS","DAYS",null,"DAYS",null,null,null,null,null,"DAYS","DAYS",null,null,null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS","DAYS",null,null,null,null,null,null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS",null,null,null,null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,"DAYS",null,"DAYS",null,null,null,null,null,null,"DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS",null,null,null,null,null,null,null,null,null,null,"DAYS",null,null,null,null,null,null,"DAYS","DAYS",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"DAYS","DAYS",null,null,null,"DAYS","DAYS",null,null,null,null,null,"DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS",null,null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS",null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS",null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS",null,null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,"DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS",null,null,"DAYS",null,null,null,"DAYS",null,null,"DAYS","DAYS",null,null,null,null,"DAYS","DAYS","DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS",null,null,null,"DAYS","DAYS",null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS",null,"DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,"DAYS",null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS",null,"DAYS","DAYS",null,"DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS",null,null,null,null,null,"DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS",null,null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,"DAYS","DAYS",null,null,"DAYS",null,null,"DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS",null,null,"DAYS","DAYS",null,null,"DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,"DAYS","DAYS",null,null,"DAYS","DAYS",null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS",null,"DAYS",null,"DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,"DAYS","DAYS",null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,"DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS",null,null,null,"DAYS",null,null,"DAYS","DAYS",null,null,"DAYS","DAYS",null,null,"DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS",null,null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS",null,null,null,null,null,null,"DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,"DAYS",null,"DAYS","DAYS",null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,null,null,null,null,null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS",null,null,null,null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,null,"DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,null,"DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,null,"DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS","DAYS",null,"DAYS","DAYS","DAYS","DAYS"],[null,null,null,null,null,null,null,null,"2013-08-01","2014-07-15","2014-07-15",null,null,"2013-02-25","2013-03-09",null,"2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15","2014-01-15",null,null,null,null,null,null,null,null,null,null,null,null,"2012-12-13","2012-12-13","2012-12-13","2012-12-13","2012-12-13",null,"2014-03-01","2014-03-01","2014-03-01","2014-03-01","2014-03-01","2014-03-01","2014-03-01","2014-03-01","2012-11-12","2012-11-12","2012-11-12","2012-11-12",null,null,null,"2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-03","2013-06-26","2013-06-26",null,null,null,null,"2013-09-05","2013-09-05","2013-09-05","2013-09-05","2013-09-05","2014-02-08","2013-10-08","2013-10-08",null,null,null,null,null,"2013-02-25","2013-03-18","2013-03-18","2013-03-18","2013-12-09",null,null,null,null,"2013-03-02","2013-03-02","2013-03-02","2013-03-02",null,null,null,"2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2012-08-07","2013-02-23",null,null,null,null,"2012-11-28","2012-11-28","2012-11-28","2012-11-28","2013-01-12",null,null,null,null,null,null,"2014-01-25","2014-01-25","2014-01-25","2014-01-25",null,null,"2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22","2014-02-22",null,"2014-02-11","2014-02-11","2014-02-11","2014-02-11",null,"2013-04-10","2013-04-10","2013-04-10","2013-04-10","2013-04-10",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15","2013-09-15",null,null,"2014-06-05","2014-06-05","2014-06-05","2014-06-05","2014-06-05","2014-06-05","2014-06-05","2012-10-16","2012-10-16","2012-10-16","2012-10-16","2013-01-18","2013-01-18","2013-01-26","2013-01-26","2013-01-26",null,null,"2013-08-05",null,null,"2013-06-12","2013-06-12","2013-06-12","2013-06-12",null,null,"2013-02-19","2013-02-19","2013-02-19","2013-02-19","2013-02-19","2013-02-19","2013-02-19","2013-02-19","2013-02-19","2013-02-19",null,null,"2012-11-11","2012-11-11","2013-10-07","2013-10-07","2013-10-07","2013-10-07","2013-10-07","2013-10-07","2013-11-09","2013-01-18","2013-01-18","2013-01-18","2013-01-18","2013-01-18","2013-01-18","2013-01-18","2013-01-18",null,null,null,null,null,null,"2013-08-08","2013-08-08","2013-08-08","2013-08-08",null,null,null,"2013-11-06","2013-11-06","2013-11-06","2013-11-06","2013-11-06","2013-11-06",null,null,null,null,"2012-09-16","2012-09-16",null,null,"2013-03-29","2013-03-29","2013-03-29","2013-03-29","2013-03-29","2013-03-29","2013-03-29","2013-03-10","2013-03-10","2013-03-10","2013-03-10",null,"2013-10-31","2013-10-31","2013-10-31","2013-10-31","2013-03-30","2013-03-30","2013-03-30",null,null,"2012-08-01","2012-08-01","2012-08-01","2012-08-01","2012-08-01","2013-12-05",null,null,null,"2013-03-01","2013-03-01","2013-03-01","2013-03-01","2013-03-13",null,null,null,null,"2012-12-13","2012-12-13","2012-12-13","2014-03-26","2014-03-26","2013-01-28","2013-02-21","2013-02-21",null,null,null,"2013-09-14","2013-09-14","2013-09-14",null,"2014-03-08","2014-03-08","2014-03-08","2014-03-08","2014-03-08","2014-03-08","2013-10-19","2013-10-19","2013-10-19","2013-10-19","2013-10-19","2013-10-19","2013-10-19","2013-10-19",null,null,"2013-10-11","2013-10-11","2013-10-24","2013-10-24",null,null,null,"2013-11-07","2013-11-07","2013-11-07","2013-11-07","2013-11-07","2013-11-07","2013-12-22",null,"2014-02-09","2013-03-30","2013-03-30","2013-03-30","2013-03-30","2013-02-03","2013-02-03","2013-02-03","2013-02-03","2013-02-03","2013-12-15","2013-12-15","2013-12-15","2013-12-15","2013-12-15","2013-12-15","2013-11-14","2013-11-14",null,null,"2012-10-04","2012-12-03",null,"2013-09-05",null,null,null,null,"2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-10-27","2013-12-06","2013-12-06","2013-07-23","2013-07-23","2013-07-23","2013-07-23","2014-05-07","2014-07-04",null,null,null,"2013-12-23","2013-12-23","2013-12-23","2013-10-26","2013-10-26","2013-10-26","2013-10-26",null,"2012-12-01","2014-05-25","2013-12-18","2013-08-19","2014-01-26",null,null,"2014-01-31","2013-12-15","2013-12-15","2013-12-15",null,"2013-10-29","2013-12-30","2013-12-30","2013-12-30","2013-12-30",null,"2013-11-22","2013-03-24","2013-03-24","2013-08-25","2013-08-25",null,null,null,"2012-09-27",null,null,null,null,null,"2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2014-01-14","2013-05-28","2013-06-18",null,null,null,null,null,null,null,null,null,null,"2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12",null,"2014-01-01","2014-01-01","2014-01-01","2014-01-01","2013-02-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2012-11-05","2012-11-05","2012-11-05","2012-11-05","2012-11-05","2012-11-05",null,"2014-03-21",null,"2014-01-18","2014-01-18","2014-01-18","2014-01-18","2014-01-18","2014-01-18","2014-01-18","2014-01-18","2013-02-22","2013-02-22",null,null,null,"2012-11-08",null,null,null,null,null,null,"2013-05-21","2013-05-21",null,null,"2013-02-21","2013-02-21","2013-02-21","2013-02-21","2013-02-21","2013-02-21","2013-02-21","2013-02-21","2013-02-21","2013-02-21",null,"2013-09-23","2013-09-23","2013-09-23","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-11-27","2013-02-08","2013-02-08","2013-05-03","2013-09-05","2014-02-15",null,"2012-12-21","2012-12-21","2012-12-21","2012-12-21","2012-12-21","2012-12-21","2013-05-04","2013-05-04","2013-05-04","2013-05-04","2013-05-04","2013-05-04","2013-05-04","2013-05-04",null,null,"2013-08-19","2013-07-18","2013-07-18","2013-08-28",null,"2014-06-16","2014-06-16",null,"2013-11-23","2013-11-23","2013-11-23","2013-11-23","2013-11-23","2013-11-23",null,null,"2012-08-18","2012-08-28","2012-12-15","2012-12-15","2012-12-15","2012-12-15","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-01-09","2013-06-12","2013-06-12","2014-02-01","2014-04-27","2014-04-26","2014-04-26",null,null,"2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-11-12","2013-01-31","2013-01-31","2013-01-31","2013-01-31","2013-01-31","2013-01-31","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17","2013-08-17",null,"2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19",null,"2013-05-29","2013-05-29","2013-05-29","2013-05-29","2013-05-29","2013-05-29","2013-05-29",null,null,"2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-02-10","2013-04-08","2013-04-08","2013-04-08","2013-04-08","2013-04-08","2013-04-08","2013-08-02","2014-02-19","2014-02-19","2014-02-19","2014-02-19","2014-02-19","2014-02-19","2014-02-19","2014-02-19","2014-02-19",null,null,null,null,null,null,null,null,null,null,"2013-07-01","2013-07-01","2013-07-01","2013-07-01","2013-07-01","2014-05-10","2014-10-19","2013-04-18","2013-04-18","2013-04-18","2013-04-18","2013-04-18","2013-04-18","2013-08-25","2013-08-25",null,"2013-01-05","2013-03-07",null,null,null,"2013-02-22","2013-02-22","2013-02-22","2013-02-22","2013-02-22","2013-02-22","2013-02-22","2013-02-22","2013-07-27","2013-07-27","2013-07-27",null,"2013-10-11","2013-10-11","2014-03-14","2014-03-14","2014-03-14","2014-03-14","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-06-17","2013-08-13","2013-08-13","2013-01-09","2013-01-09","2013-01-09","2013-01-09",null,null,null,"2013-01-15","2013-01-15","2013-01-15","2013-01-15","2013-01-15","2012-09-22","2012-09-22","2012-09-22","2012-09-22","2013-01-22","2013-01-22",null,null,"2013-12-01","2013-12-01","2013-12-01","2013-08-01",null,null,"2013-10-25","2013-11-13",null,null,null,"2012-10-16","2012-10-16","2014-04-12","2014-04-12","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-11-30","2013-11-30","2012-11-24","2012-10-10","2012-10-10","2012-10-10","2014-01-11","2014-01-11","2014-01-11","2014-01-11","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-06-27","2013-10-11","2013-10-11",null,null,"2012-11-21","2012-11-21","2012-11-21","2012-11-21","2012-11-21","2013-01-07","2013-01-07","2013-01-07","2013-01-07","2013-01-07","2013-01-07","2013-02-26","2012-12-30","2012-12-30","2012-12-30","2012-12-30","2013-05-09","2013-05-09",null,"2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2013-03-13","2012-11-25","2012-11-25","2012-10-01","2012-10-01","2012-10-01","2013-11-06","2013-11-06","2013-11-06","2013-11-06","2012-11-12","2012-11-12","2012-11-12","2012-11-12","2012-11-12","2012-11-12","2013-02-18","2013-02-18","2013-01-24","2013-01-24","2013-01-24","2013-01-24",null,null,null,null,null,"2013-04-17","2013-04-29",null,null,null,"2012-08-12",null,null,"2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-04-17","2013-05-30","2013-05-30",null,null,"2013-02-05",null,null,"2013-09-30","2013-09-30","2013-09-30","2013-09-30","2014-02-28","2014-04-14","2013-06-14","2013-06-14","2013-06-14","2013-07-01",null,null,"2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24","2013-08-24",null,null,null,"2013-06-14","2012-10-03","2012-10-03",null,"2014-02-10","2014-02-10","2014-02-10","2014-02-10","2014-02-10","2014-02-01","2014-02-01","2014-02-01","2014-02-01","2014-05-01","2014-05-01","2014-05-01","2014-05-01","2014-05-01","2014-05-01","2014-05-01",null,null,"2013-08-22","2013-08-22","2013-08-22","2013-08-22","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09","2013-05-09",null,null,"2013-12-17","2013-12-17","2013-12-17","2013-12-17","2013-12-17","2013-12-17","2014-05-27","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19","2013-03-19",null,"2013-05-07","2013-05-07","2013-05-07","2012-12-02","2013-03-05","2014-02-26","2014-02-26","2014-04-21","2014-04-21","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2013-07-07","2012-07-23","2012-07-23","2012-07-23",null,null,"2014-04-18","2014-04-18","2014-04-18","2014-04-18","2014-04-18","2013-05-10","2013-05-10","2013-05-10",null,"2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-05-23","2013-06-22","2013-06-22","2013-06-22","2013-01-06","2013-01-06","2013-01-06","2014-04-05","2014-04-05","2014-04-05","2014-04-05","2014-04-05","2013-02-26","2013-02-26","2013-02-15","2013-02-15","2013-02-15","2013-02-15","2013-10-16","2013-10-16","2013-10-16","2013-10-16","2013-04-19","2013-04-19","2012-10-29","2012-10-29","2012-10-29","2012-10-29","2012-10-29",null,"2012-10-24","2012-10-24","2012-10-24","2012-10-24",null,"2013-03-06","2013-04-22","2013-04-22","2013-04-22","2013-04-22","2013-04-22","2013-04-22","2013-09-12","2014-05-28","2014-05-28","2014-05-28","2014-05-28","2013-07-17","2013-07-17",null,null,null,null,null,null,"2013-05-19","2013-05-19","2013-05-19","2013-05-19","2013-05-19","2013-05-19","2014-02-06","2013-12-31","2013-12-31","2013-12-31","2013-12-31","2013-12-31",null,null,null,null,null,null,null,null,null,null,null,null,"2014-01-28","2014-01-28","2014-01-28","2014-01-28","2014-01-28","2014-01-28","2014-01-28",null,null,"2014-02-10","2014-02-10","2014-02-10","2014-02-10","2014-02-10","2014-02-10",null,null,null,null,"2013-02-04","2013-02-04",null,"2014-01-03","2014-01-24","2014-01-24","2014-01-24","2014-01-24","2014-01-24",null,"2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15","2013-05-15",null,null,"2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-09-16","2013-07-16","2013-07-16","2013-07-16","2013-07-16","2012-10-03","2012-10-03","2012-10-03","2013-03-03","2013-03-03","2013-03-03","2013-03-03","2013-03-03","2013-03-03","2013-06-02","2013-06-02","2013-11-03","2013-02-02","2013-02-02","2013-02-02","2013-02-02","2013-02-02","2013-02-02","2013-02-02","2013-02-02","2013-02-02","2013-07-29","2013-09-29","2013-09-29","2013-09-29","2013-09-29","2013-10-12","2013-10-12","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03","2013-10-03",null,"2013-07-25","2013-07-25","2013-07-25","2013-07-25","2013-07-25","2013-07-25","2013-07-25","2013-07-25","2013-02-14","2013-02-14","2013-02-14","2013-02-14",null,null,null,"2013-03-16","2013-03-16","2013-03-16","2013-03-16","2013-03-16","2013-03-16","2013-03-16","2013-03-16","2013-08-15","2013-08-15","2013-05-08","2013-05-08","2013-05-08","2013-05-08","2013-05-08","2013-05-08",null,null,"2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31","2012-12-31"],["Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,null,null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y",null,"Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y"],["Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,null,null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y","Y","Y","Y","Y",null,null,"Y","Y",null,"Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,null,null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y",null,"Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y","Y"],["Y",null,null,null,"Y",null,null,"Y",null,"Y",null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,"Y",null,null,null,null,"Y",null,null,null,"Y",null,null,"Y",null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,"Y",null,"Y",null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y","Y",null,"Y",null,null,null,null,null,null,null,null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,null,null,"Y",null,null,"Y",null,"Y",null,null,null,null,"Y","Y",null,null,null,null,null,null,"Y","Y",null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,"Y",null,"Y",null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,"Y","Y",null,null,"Y","Y",null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,"Y",null,null,"Y",null,null,null,null,null,null,"Y",null,"Y","Y","Y","Y","Y","Y","Y",null,"Y","Y",null,null,null,"Y",null,null,null,"Y","Y",null,"Y",null,null,null,null,null,null,"Y","Y",null,"Y",null,null,null,null,null,null,null,null,null,"Y","Y",null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,"Y","Y",null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,"Y","Y",null,null,null,"Y",null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,"Y",null,null,"Y",null,null,"Y","Y",null,null,null,null,null,"Y",null,null,null,null,"Y",null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,"Y",null,"Y",null,"Y","Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,"Y",null,null,"Y",null,"Y",null,null,null,null,null,"Y",null,"Y",null,null,null,null,"Y","Y","Y",null,null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,"Y",null,null,null,null,null,null,"Y",null,"Y",null,null,"Y",null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,"Y",null,null,null,null,null,"Y",null,null,"Y",null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,"Y",null,null,null,null,"Y",null,null,"Y",null,null,null,null,null,null,null,null,null,"Y","Y",null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,"Y","Y",null,null,"Y","Y",null,null,"Y",null,null,null,null,"Y",null,null,"Y",null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,"Y","Y",null,null,null,null,null,"Y",null,"Y",null,null,null,null,"Y","Y",null,null,null,null,"Y","Y",null,null,null,null,"Y",null,null,null,"Y","Y",null,"Y",null,null,null,null,null,"Y",null,null,"Y",null,"Y",null,null,"Y",null,null,null,null,"Y",null,null,null,null,null,"Y",null,"Y",null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,null,"Y","Y",null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,"Y",null,null,null,null,"Y",null,"Y",null,null,null,null,null,"Y",null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,"Y",null,"Y",null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null,null,null,null,"Y","Y",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,"Y",null,null,null,null,null,null,null,null],[null,null,null,"Dermatologic events","Dermatologic events","Dermatologic events",null,"Dermatologic events",null,null,null,null,null,null,null,"Dermatologic events",null,"Dermatologic events",null,null,"Dermatologic events","Dermatologic events",null,null,"Dermatologic events",null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,"Dermatologic events",null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,"Dermatologic events",null,null,null,"Dermatologic events",null,"Dermatologic events",null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,"Dermatologic events","Dermatologic events",null,null,null,null,"Dermatologic events","Dermatologic events","Dermatologic events",null,null,null,"Dermatologic events",null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,"Dermatologic events",null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Dermatologic events",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>STUDYID<\/th>\n      <th>USUBJID<\/th>\n      <th>SUBJID<\/th>\n      <th>SITEID<\/th>\n      <th>REGION1<\/th>\n      <th>COUNTRY<\/th>\n      <th>ETHNIC<\/th>\n      <th>AGE<\/th>\n      <th>AGEU<\/th>\n      <th>SEX<\/th>\n      <th>RACE<\/th>\n      <th>TRT01P<\/th>\n      <th>TRT01A<\/th>\n      <th>TRTSDT<\/th>\n      <th>TRTSDTM<\/th>\n      <th>TRTEDT<\/th>\n      <th>SAFFL<\/th>\n      <th>DTHDT<\/th>\n      <th>ACTARM<\/th>\n      <th>AESEQ<\/th>\n      <th>AETERM<\/th>\n      <th>AEDECOD<\/th>\n      <th>AESEV<\/th>\n      <th>AESER<\/th>\n      <th>AEACN<\/th>\n      <th>AEREL<\/th>\n      <th>AEOUT<\/th>\n      <th>AESTDTC<\/th>\n      <th>AEENDTC<\/th>\n      <th>ASTDT<\/th>\n      <th>ASTDTF<\/th>\n      <th>ASTDY<\/th>\n      <th>AENDT<\/th>\n      <th>AENDTF<\/th>\n      <th>AENDY<\/th>\n      <th>ADURN<\/th>\n      <th>ADURU<\/th>\n      <th>LDOSEDT<\/th>\n      <th>TRTEMFL<\/th>\n      <th>ONTRTFL<\/th>\n      <th>AOCCIFL<\/th>\n      <th>CQ01NAM<\/th>\n      <th>SMQ02NAM<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[8,20,32,35,36]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"STUDYID","targets":1},{"name":"USUBJID","targets":2},{"name":"SUBJID","targets":3},{"name":"SITEID","targets":4},{"name":"REGION1","targets":5},{"name":"COUNTRY","targets":6},{"name":"ETHNIC","targets":7},{"name":"AGE","targets":8},{"name":"AGEU","targets":9},{"name":"SEX","targets":10},{"name":"RACE","targets":11},{"name":"TRT01P","targets":12},{"name":"TRT01A","targets":13},{"name":"TRTSDT","targets":14},{"name":"TRTSDTM","targets":15},{"name":"TRTEDT","targets":16},{"name":"SAFFL","targets":17},{"name":"DTHDT","targets":18},{"name":"ACTARM","targets":19},{"name":"AESEQ","targets":20},{"name":"AETERM","targets":21},{"name":"AEDECOD","targets":22},{"name":"AESEV","targets":23},{"name":"AESER","targets":24},{"name":"AEACN","targets":25},{"name":"AEREL","targets":26},{"name":"AEOUT","targets":27},{"name":"AESTDTC","targets":28},{"name":"AEENDTC","targets":29},{"name":"ASTDT","targets":30},{"name":"ASTDTF","targets":31},{"name":"ASTDY","targets":32},{"name":"AENDT","targets":33},{"name":"AENDTF","targets":34},{"name":"AENDY","targets":35},{"name":"ADURN","targets":36},{"name":"ADURU","targets":37},{"name":"LDOSEDT","targets":38},{"name":"TRTEMFL","targets":39},{"name":"ONTRTFL","targets":40},{"name":"AOCCIFL","targets":41},{"name":"CQ01NAM","targets":42},{"name":"SMQ02NAM","targets":43}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>
</div>
</div>
</section></section><section id="adpc" class="level2"><h2 class="anchored" data-anchor-id="adpc">ADPC</h2>
<p>약물동력학(PK) 데이터를 분석하기 위한 핵심 데이터셋으로, 시간에 따른 혈중 약물 농도를 기록한다.</p>
<p>목적: <br> - 시간에 따른 약물 농도(PK 데이터)를 분석하여 약물 흡수, 분포, 대사, 배설(ADME) 평가. <br><br> 구조: <br> - 각 행은 특정 시간점(NTIM)에서의 환자별 농도(DV)를 기록하며, USUBJID와 시간점으로 구분. <br><br> 주요 활용: <br> - PK 파라미터(예: AUC, Cmax, Tmax) 계산 및 모델링. <br></p>
<p><strong>주요 변수 (Columns)</strong></p>
<table class="caption-top table">
<colgroup>
<col style="width: 23%">
<col style="width: 30%">
<col style="width: 46%">
</colgroup>
<thead><tr class="header">
<th>Category</th>
<th>Variables</th>
<th>Description</th>
</tr></thead>
<tbody>
<tr class="odd">
<td><strong>Identifiers</strong></td>
<td>• <code>USUBJID</code><br>• <code>SUBJID</code><br>• <code>VISIT</code><br>• <code>VISITNUM</code>
</td>
<td>• 고유 환자 ID<br>• 대상자 ID<br>• 방문 이름<br>• 방문 번호</td>
</tr>
<tr class="even">
<td><strong>Time Info</strong></td>
<td>• <code>NTIM</code><br>• <code>TIME</code><br>• <code>NOM_*</code>
</td>
<td>• 실제 시간(hr)<br>• 계획 시간<br>• 명목 시간(일/주 단위)</td>
</tr>
<tr class="odd">
<td><strong>Concentration</strong></td>
<td>• <code>DV</code><br>• <code>LLOQ</code><br>• <code>BLQ</code>
</td>
<td>• 관측 농도(ng/mL)<br>• 정량 한계<br>• 한계 미만 여부(Y/N)</td>
</tr>
<tr class="even">
<td><strong>Dose Info</strong></td>
<td>• <code>DOSE</code><br>• <code>DOSEUNIT</code><br>• <code>ROUTE</code>
</td>
<td>• 투여 용량<br>• 단위(mg/kg)<br>• 투여 경로(IV/PO)</td>
</tr>
<tr class="odd">
<td><strong>PK Parameters</strong></td>
<td>• <code>AUC</code><br>• <code>CMAX</code><br>• <code>TMAX</code>
</td>
<td>• 농도-시간 곡선 아래 면적<br>• 최대 농도<br>• 최대 농도 도달 시간</td>
</tr>
<tr class="even">
<td><strong>Standards</strong></td>
<td>All variables</td>
<td>CDISC ADaM 표준 준수</td>
</tr>
</tbody>
</table></section><section id="adppk" class="level2"><h2 class="anchored" data-anchor-id="adppk">ADPPK</h2>
<p>DPPK는 약물동력학(PK) 파라미터(예: AUC, Cmax, Tmax)를 분석하기 위한 데이터셋으로, ADPC에서 계산된 PK 파라미터를 주로 다룬다.</p>
<p>목적: <br> - PK 파라미터를 조직화하여 약물 노출량, 반감기, 청소율 등을 분석. <br><br> 구조: <br> - 각 행은 환자별 PK 파라미터를 포함하며, USUBJID와 <strong>파라미터 종류(PARAM/PARAMCD)</strong>로 구분. <br><br> 차이점: <br> - <code>ADPC</code>: 시간별 농도 데이터 (Raw PK 데이터). <br> - <code>ADPPK</code>: 파생된 PK 파라미터 (예: AUCINF, CL). <br></p>
<p><strong>주요 변수 (Columns) </strong></p>
<table class="caption-top table">
<colgroup>
<col style="width: 23%">
<col style="width: 30%">
<col style="width: 46%">
</colgroup>
<thead><tr class="header">
<th>Category</th>
<th>Variables</th>
<th>Description</th>
</tr></thead>
<tbody>
<tr class="odd">
<td><strong>Identifiers</strong></td>
<td>• <code>USUBJID</code><br>• <code>SUBJID</code><br>• <code>STUDYID</code>
</td>
<td>• 고유 환자 ID<br>• 대상자 ID<br>• 연구 ID</td>
</tr>
<tr class="even">
<td><strong>PK Parameters</strong></td>
<td>• <code>PARAM</code>/<code>PARAMCD</code><br>• <code>AVAL</code><br>• <code>AVALU</code>
</td>
<td>• 파라미터 이름/코드<br>• 실제 값<br>• 단위</td>
</tr>
<tr class="odd">
<td><strong>Dose Info</strong></td>
<td>• <code>DOSE</code><br>• <code>DOSU</code>
</td>
<td>• 투여 용량<br>• 용량 단위</td>
</tr>
<tr class="even">
<td><strong>Analysis</strong></td>
<td>• <code>ANALYZT</code><br>• <code>METHOD</code>
</td>
<td>• 분석 방법<br>• 계산 알고리즘</td>
</tr>
<tr class="odd">
<td><strong>Standards</strong></td>
<td>All variables</td>
<td>CDISC ADaM 표준 준수</td>
</tr>
</tbody>
</table></section></section><section id="tlg" class="level1"><h1>TLG</h1>
<ul>
<li>talbes, listings, Graphs</li>
<li><strong>demographic table &amp; adverse event table</strong></li>
</ul>
<section id="demopragphic-table" class="level2"><h2 class="anchored" data-anchor-id="demopragphic-table">demopragphic table</h2>
<ul>
<li>
<code>ADSL</code> 데이터를 이용하여 작성한다.</li>
<li>
<a href="https://insightsengineering.github.io/tern/">tern</a>: 임상시험 보고용 테이블 및 그래프 생성을 위한 패키지</li>
</ul>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb38" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://insightsengineering.github.io/tern/">tern</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>   </span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1. 결측값 명시적 처리</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">df_explicit_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 2. 테이블 레이아웃 정의</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lyt</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">basic_table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>show_colcounts <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">split_cols_by</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ACTARM"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 치료군별 컬럼 분할</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_overall_col</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"All Patients"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>   <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 전체 집단 컬럼 추가</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">analyze_vars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AGE"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AGEGR1"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"SEX"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RACE"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>    var_labels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Age (yr)"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Age group"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Sex"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Race"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 3. 테이블 빌드</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">result</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">build_table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lyt</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">print</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>                                       Placebo     Screen Failure   Xanomeline High Dose   Xanomeline Low Dose   All Patients
                                       (N=86)          (N=52)              (N=72)                (N=96)            (N=306)   
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
Age (yr)                                                                                                                     
  n                                      86              52                  72                    96                306     
  Mean (SD)                          75.2 (8.6)      75.1 (9.7)          73.8 (7.9)            76.0 (8.1)         75.1 (8.5) 
  Median                                76.0            76.0                75.5                  78.0               77.0    
  Min - Max                          52.0 - 89.0    50.0 - 89.0         56.0 - 88.0            51.0 - 88.0       50.0 - 89.0 
Age group                                                                                                                    
  n                                      86              52                  72                    96                306     
  &gt;64                                72 (83.7%)      43 (82.7%)          61 (84.7%)            88 (91.7%)        264 (86.3%) 
  18-64                              14 (16.3%)      9 (17.3%)           11 (15.3%)             8 (8.3%)          42 (13.7%) 
Sex                                                                                                                          
  n                                      86              52                  72                    96                306     
  F                                  53 (61.6%)      36 (69.2%)          35 (48.6%)            55 (57.3%)        179 (58.5%) 
  M                                  33 (38.4%)      16 (30.8%)          37 (51.4%)            41 (42.7%)        127 (41.5%) 
Race                                                                                                                         
  n                                      86              52                  72                    96                306     
  AMERICAN INDIAN OR ALASKA NATIVE        0           1 (1.9%)            1 (1.4%)                  0              2 (0.7%)  
  ASIAN                                   0           2 (3.8%)               0                      0              2 (0.7%)  
  BLACK OR AFRICAN AMERICAN           8 (9.3%)       6 (11.5%)           9 (12.5%)              6 (6.2%)          29 (9.5%)  
  WHITE                              78 (90.7%)      43 (82.7%)          62 (86.1%)            90 (93.8%)        273 (89.2%) </code></pre>
</div>
</div>
</section><section id="adverse-event-table" class="level2"><h2 class="anchored" data-anchor-id="adverse-event-table">adverse event table</h2>
<ul>
<li>
<code>ADSL</code>, <code>ADAE</code> 데이터를 이용하여 작성한다.</li>
</ul>
<p>입력 데이터: <br> &nbsp;&nbsp; <code>adsl</code> (기본 환자 정보) <br> &nbsp;&nbsp; <code>adae</code> (이상반응 데이터) <br><br> 출력: <br> &nbsp;&nbsp; 치료군별/신체계별 AE 발생 현황을 보여주는 분석용 테이블 <br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb40" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://pharmaverse.github.io/pharmaverseadam/">pharmaverseadam</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://insightsengineering.github.io/tern/">tern</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 데이터 전처리</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 결측값 처리</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">df_explicit_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">df_explicit_na</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 라벨 추가 및 필터링</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">var_relabel</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      AEBODSYS <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MedDRA System Organ Class"</span>,  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 신체계 라벨</span></span>
<span>      AEDECOD <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MedDRA Preferred Term"</span>        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># PT 라벨</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/filter.html">filter</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">SAFFL</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Y"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 안전성 평가 대상만 선택</span></span>
<span>        <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 분할 함수 정의</span></span>
<span>          <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">split_fun</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">drop_split_levels</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 테이블 레이아웃 정의.</span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 테이블 구조 설계</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lyt</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">basic_table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>show_colcounts <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 1. 컬럼 분할 (치료군 기준)</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">split_cols_by</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ACTARM"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span>  </span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">add_overall_col</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>label <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"All Patients"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 2. 전체 환자/이벤트 수 요약</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">analyze_num_patients</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>,</span>
<span>      .stats <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"unique"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nonunique"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      .labels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>        unique <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Total number of patients with at least one adverse event"</span>,</span>
<span>        nonunique <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Overall total number of events"</span></span>
<span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 3. 행 분할 (신체계 기준)</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">split_rows_by</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AEBODSYS"</span>,</span>
<span>      child_labels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"visible"</span>,</span>
<span>      nested <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>,</span>
<span>      split_fun <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">split_fun</span>,</span>
<span>      label_pos <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"topleft"</span>,</span>
<span>      split_label <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">obj_label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AEBODSYS</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 4. 환자/이벤트 수 요약</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">summarize_num_patients</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      var <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"USUBJID"</span>,</span>
<span>      .stats <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"unique"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"nonunique"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>      .labels <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>        unique <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Total number of patients with at least one adverse event"</span>,</span>
<span>        nonunique <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Total number of events"</span></span>
<span>      <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 5. 특정 반응(PT) 발생 횟수 계산</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">count_occurrences</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>      vars <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AEDECOD"</span>,</span>
<span>      .indent_mods <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1L</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">%&gt;%</span></span>
<span>    </span>
<span>    <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 6. 라벨 추가</span></span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">append_varlabels</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AEDECOD"</span>, indent <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1L</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">result</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">build_table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lyt</span>, df <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adae</span>, alt_counts_df <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">adsl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>; <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">result</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>MedDRA System Organ Class                                              Placebo     Xanomeline High Dose   Xanomeline Low Dose   All Patients
  MedDRA Preferred Term                                                 (N=86)            (N=72)                (N=96)            (N=306)   
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
Total number of patients with at least one adverse event              69 (80.2%)        70 (97.2%)            86 (89.6%)        225 (73.5%) 
Overall total number of events                                           301               436                    454               1191    
CARDIAC DISORDERS                                                                                                                           
  Total number of patients with at least one adverse event            13 (15.1%)        15 (20.8%)            16 (16.7%)         44 (14.4%) 
  Total number of events                                                  27                30                    34                 91     
  ATRIAL FIBRILLATION                                                  1 (1.2%)          2 (2.8%)              2 (2.1%)           5 (1.6%)  
  ATRIAL FLUTTER                                                          0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  ATRIAL HYPERTROPHY                                                   1 (1.2%)             0                      0              1 (0.3%)  
  ATRIOVENTRICULAR BLOCK FIRST DEGREE                                  1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  ATRIOVENTRICULAR BLOCK SECOND DEGREE                                 2 (2.3%)          1 (1.4%)              2 (2.1%)           5 (1.6%)  
  BRADYCARDIA                                                          1 (1.2%)             0                      0              1 (0.3%)  
  BUNDLE BRANCH BLOCK LEFT                                             1 (1.2%)             0                      0              1 (0.3%)  
  BUNDLE BRANCH BLOCK RIGHT                                            1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  CARDIAC DISORDER                                                        0              1 (1.4%)                  0              1 (0.3%)  
  CARDIAC FAILURE CONGESTIVE                                           1 (1.2%)             0                      0              1 (0.3%)  
  MYOCARDIAL INFARCTION                                                4 (4.7%)          4 (5.6%)              2 (2.1%)          10 (3.3%)  
  PALPITATIONS                                                            0                 0                  2 (2.1%)           2 (0.7%)  
  SINUS ARRHYTHMIA                                                     1 (1.2%)             0                      0              1 (0.3%)  
  SINUS BRADYCARDIA                                                    2 (2.3%)         8 (11.1%)              7 (7.3%)          17 (5.6%)  
  SUPRAVENTRICULAR EXTRASYSTOLES                                       1 (1.2%)          1 (1.4%)              1 (1.0%)           3 (1.0%)  
  SUPRAVENTRICULAR TACHYCARDIA                                            0                 0                  1 (1.0%)           1 (0.3%)  
  TACHYCARDIA                                                          1 (1.2%)             0                      0              1 (0.3%)  
  VENTRICULAR EXTRASYSTOLES                                               0              1 (1.4%)              2 (2.1%)           3 (1.0%)  
  VENTRICULAR HYPERTROPHY                                              1 (1.2%)             0                      0              1 (0.3%)  
  WOLFF-PARKINSON-WHITE SYNDROME                                          0                 0                  1 (1.0%)           1 (0.3%)  
CONGENITAL, FAMILIAL AND GENETIC DISORDERS                                                                                                  
  Total number of patients with at least one adverse event                0              2 (2.8%)              1 (1.0%)           3 (1.0%)  
  Total number of events                                                  0                 2                      1                 3      
  VENTRICULAR SEPTAL DEFECT                                               0              2 (2.8%)              1 (1.0%)           3 (1.0%)  
EAR AND LABYRINTH DISORDERS                                                                                                                 
  Total number of patients with at least one adverse event             1 (1.2%)          1 (1.4%)              2 (2.1%)           4 (1.3%)  
  Total number of events                                                  2                 1                      3                 6      
  CERUMEN IMPACTION                                                       0                 0                  1 (1.0%)           1 (0.3%)  
  EAR PAIN                                                             1 (1.2%)             0                      0              1 (0.3%)  
  TINNITUS                                                                0                 0                  1 (1.0%)           1 (0.3%)  
  VERTIGO                                                                 0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
EYE DISORDERS                                                                                                                               
  Total number of patients with at least one adverse event             4 (4.7%)          1 (1.4%)              2 (2.1%)           7 (2.3%)  
  Total number of events                                                  8                 2                      2                 12     
  CONJUNCTIVAL HAEMORRHAGE                                                0                 0                  1 (1.0%)           1 (0.3%)  
  CONJUNCTIVITIS                                                       2 (2.3%)             0                      0              2 (0.7%)  
  EYE ALLERGY                                                          1 (1.2%)             0                      0              1 (0.3%)  
  EYE PRURITUS                                                         1 (1.2%)             0                      0              1 (0.3%)  
  EYE SWELLING                                                         1 (1.2%)             0                      0              1 (0.3%)  
  GLAUCOMA                                                             1 (1.2%)             0                      0              1 (0.3%)  
  VISION BLURRED                                                          0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
GASTROINTESTINAL DISORDERS                                                                                                                  
  Total number of patients with at least one adverse event            17 (19.8%)        20 (27.8%)            16 (16.7%)         53 (17.3%) 
  Total number of events                                                  26                35                    26                 87     
  ABDOMINAL DISCOMFORT                                                    0              1 (1.4%)                  0              1 (0.3%)  
  ABDOMINAL PAIN                                                       1 (1.2%)          1 (1.4%)              3 (3.1%)           5 (1.6%)  
  CONSTIPATION                                                         1 (1.2%)             0                      0              1 (0.3%)  
  DIARRHOEA                                                           9 (10.5%)          3 (4.2%)              6 (6.2%)          18 (5.9%)  
  DYSPEPSIA                                                            1 (1.2%)          1 (1.4%)              1 (1.0%)           3 (1.0%)  
  DYSPHAGIA                                                               0                 0                  1 (1.0%)           1 (0.3%)  
  FLATULENCE                                                           1 (1.2%)             0                      0              1 (0.3%)  
  GASTROINTESTINAL HAEMORRHAGE                                            0              1 (1.4%)                  0              1 (0.3%)  
  GASTROOESOPHAGEAL REFLUX DISEASE                                     1 (1.2%)             0                      0              1 (0.3%)  
  GLOSSITIS                                                            1 (1.2%)             0                      0              1 (0.3%)  
  HIATUS HERNIA                                                        1 (1.2%)             0                      0              1 (0.3%)  
  NAUSEA                                                               3 (3.5%)          6 (8.3%)              3 (3.1%)          12 (3.9%)  
  RECTAL HAEMORRHAGE                                                      0                 0                  1 (1.0%)           1 (0.3%)  
  SALIVARY HYPERSECRETION                                                 0              4 (5.6%)                  0              4 (1.3%)  
  STOMACH DISCOMFORT                                                      0              1 (1.4%)                  0              1 (0.3%)  
  VOMITING                                                             3 (3.5%)          6 (8.3%)              4 (4.2%)          13 (4.2%)  
GENERAL DISORDERS AND ADMINISTRATION SITE CONDITIONS                                                                                        
  Total number of patients with at least one adverse event            21 (24.4%)        36 (50.0%)            51 (53.1%)        108 (35.3%) 
  Total number of events                                                  48               118                    126               292     
  APPLICATION SITE BLEEDING                                               0                 0                  1 (1.0%)           1 (0.3%)  
  APPLICATION SITE DERMATITIS                                          5 (5.8%)          7 (9.7%)              9 (9.4%)          21 (6.9%)  
  APPLICATION SITE DESQUAMATION                                           0                 0                  1 (1.0%)           1 (0.3%)  
  APPLICATION SITE DISCHARGE                                              0              1 (1.4%)                  0              1 (0.3%)  
  APPLICATION SITE DISCOLOURATION                                         0                 0                  1 (1.0%)           1 (0.3%)  
  APPLICATION SITE ERYTHEMA                                            3 (3.5%)         14 (19.4%)            13 (13.5%)         30 (9.8%)  
  APPLICATION SITE INDURATION                                          1 (1.2%)             0                      0              1 (0.3%)  
  APPLICATION SITE IRRITATION                                          3 (3.5%)         9 (12.5%)              9 (9.4%)          21 (6.9%)  
  APPLICATION SITE PAIN                                                   0              2 (2.8%)                  0              2 (0.7%)  
  APPLICATION SITE PERSPIRATION                                           0              2 (2.8%)                  0              2 (0.7%)  
  APPLICATION SITE PRURITUS                                            6 (7.0%)         21 (29.2%)            23 (24.0%)         50 (16.3%) 
  APPLICATION SITE REACTION                                            1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  APPLICATION SITE SWELLING                                               0              2 (2.8%)              1 (1.0%)           3 (1.0%)  
  APPLICATION SITE URTICARIA                                              0              1 (1.4%)              2 (2.1%)           3 (1.0%)  
  APPLICATION SITE VESICLES                                            1 (1.2%)          5 (6.9%)              5 (5.2%)          11 (3.6%)  
  APPLICATION SITE WARMTH                                                 0                 0                  1 (1.0%)           1 (0.3%)  
  ASTHENIA                                                             1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  CHEST DISCOMFORT                                                        0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  CHEST PAIN                                                              0              2 (2.8%)                  0              2 (0.7%)  
  CHILLS                                                               1 (1.2%)          1 (1.4%)              1 (1.0%)           3 (1.0%)  
  CYST                                                                    0                 0                  1 (1.0%)           1 (0.3%)  
  FATIGUE                                                              1 (1.2%)          5 (6.9%)              5 (5.2%)          11 (3.6%)  
  FEELING ABNORMAL                                                        0              1 (1.4%)                  0              1 (0.3%)  
  FEELING COLD                                                            0              1 (1.4%)                  0              1 (0.3%)  
  INFLAMMATION                                                            0                 0                  1 (1.0%)           1 (0.3%)  
  MALAISE                                                                 0              2 (2.8%)              1 (1.0%)           3 (1.0%)  
  OEDEMA                                                                  0                 0                  2 (2.1%)           2 (0.7%)  
  OEDEMA PERIPHERAL                                                    2 (2.3%)          2 (2.8%)              1 (1.0%)           5 (1.6%)  
  PAIN                                                                    0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  PYREXIA                                                              2 (2.3%)             0                  1 (1.0%)           3 (1.0%)  
  SECRETION DISCHARGE                                                     0                 0                  1 (1.0%)           1 (0.3%)  
  SUDDEN DEATH                                                            0                 0                  1 (1.0%)           1 (0.3%)  
  SWELLING                                                                0                 0                  1 (1.0%)           1 (0.3%)  
  ULCER                                                                   0                 0                  1 (1.0%)           1 (0.3%)  
HEPATOBILIARY DISORDERS                                                                                                                     
  Total number of patients with at least one adverse event             1 (1.2%)             0                      0              1 (0.3%)  
  Total number of events                                                  1                 0                      0                 1      
  HYPERBILIRUBINAEMIA                                                  1 (1.2%)             0                      0              1 (0.3%)  
IMMUNE SYSTEM DISORDERS                                                                                                                     
  Total number of patients with at least one adverse event                0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  Total number of events                                                  0                 1                      2                 3      
  HYPERSENSITIVITY                                                        0                 0                  1 (1.0%)           1 (0.3%)  
  SEASONAL ALLERGY                                                        0              1 (1.4%)                  0              1 (0.3%)  
INFECTIONS AND INFESTATIONS                                                                                                                 
  Total number of patients with at least one adverse event            16 (18.6%)        13 (18.1%)            10 (10.4%)         39 (12.7%) 
  Total number of events                                                  35                20                    18                 73     
  BRONCHITIS                                                           1 (1.2%)             0                      0              1 (0.3%)  
  CELLULITIS                                                              0                 0                  1 (1.0%)           1 (0.3%)  
  CERVICITIS                                                           1 (1.2%)             0                      0              1 (0.3%)  
  CYSTITIS                                                             1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  EAR INFECTION                                                        2 (2.3%)             0                      0              2 (0.7%)  
  GASTROENTERITIS VIRAL                                                1 (1.2%)             0                      0              1 (0.3%)  
  HORDEOLUM                                                               0              1 (1.4%)                  0              1 (0.3%)  
  INFLUENZA                                                            1 (1.2%)          1 (1.4%)              1 (1.0%)           3 (1.0%)  
  LOCALISED INFECTION                                                  1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  LOWER RESPIRATORY TRACT INFECTION                                       0              1 (1.4%)                  0              1 (0.3%)  
  NASOPHARYNGITIS                                                      2 (2.3%)          6 (8.3%)              4 (4.2%)          12 (3.9%)  
  ONYCHOMYCOSIS                                                           0                 0                  1 (1.0%)           1 (0.3%)  
  PNEUMONIA                                                               0                 0                  1 (1.0%)           1 (0.3%)  
  RHINITIS                                                                0              1 (1.4%)                  0              1 (0.3%)  
  UPPER RESPIRATORY TRACT INFECTION                                    6 (7.0%)          3 (4.2%)              1 (1.0%)          10 (3.3%)  
  URINARY TRACT INFECTION                                              2 (2.3%)          1 (1.4%)                  0              3 (1.0%)  
  VAGINAL MYCOSIS                                                      1 (1.2%)             0                      0              1 (0.3%)  
  VIRAL INFECTION                                                         0                 0                  1 (1.0%)           1 (0.3%)  
INJURY, POISONING AND PROCEDURAL COMPLICATIONS                                                                                              
  Total number of patients with at least one adverse event             4 (4.7%)          5 (6.9%)              5 (5.2%)          14 (4.6%)  
  Total number of events                                                  9                 8                     12                 29     
  CONTUSION                                                            1 (1.2%)          2 (2.8%)              1 (1.0%)           4 (1.3%)  
  EXCORIATION                                                          2 (2.3%)          1 (1.4%)              1 (1.0%)           4 (1.3%)  
  FACIAL BONES FRACTURE                                                   0              1 (1.4%)                  0              1 (0.3%)  
  FALL                                                                 1 (1.2%)          1 (1.4%)              2 (2.1%)           4 (1.3%)  
  HIP FRACTURE                                                         1 (1.2%)          2 (2.8%)                  0              3 (1.0%)  
  JOINT DISLOCATION                                                       0                 0                  1 (1.0%)           1 (0.3%)  
  SKIN LACERATION                                                      1 (1.2%)             0                  2 (2.1%)           3 (1.0%)  
  WOUND                                                                   0                 0                  1 (1.0%)           1 (0.3%)  
INVESTIGATIONS                                                                                                                              
  Total number of patients with at least one adverse event            10 (11.6%)         5 (6.9%)              8 (8.3%)          23 (7.5%)  
  Total number of events                                                  19                6                     15                 40     
  BIOPSY                                                                  0              1 (1.4%)                  0              1 (0.3%)  
  BIOPSY PROSTATE                                                         0              1 (1.4%)                  0              1 (0.3%)  
  BLOOD ALKALINE PHOSPHATASE INCREASED                                 1 (1.2%)             0                      0              1 (0.3%)  
  BLOOD CHOLESTEROL INCREASED                                             0              1 (1.4%)                  0              1 (0.3%)  
  BLOOD CREATINE PHOSPHOKINASE INCREASED                               1 (1.2%)             0                      0              1 (0.3%)  
  BLOOD GLUCOSE INCREASED                                                 0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  BLOOD URINE PRESENT                                                  1 (1.2%)             0                      0              1 (0.3%)  
  BODY TEMPERATURE INCREASED                                              0                 0                  1 (1.0%)           1 (0.3%)  
  CYSTOSCOPY                                                           1 (1.2%)             0                      0              1 (0.3%)  
  ELECTROCARDIOGRAM ST SEGMENT DEPRESSION                              4 (4.7%)             0                  1 (1.0%)           5 (1.6%)  
  ELECTROCARDIOGRAM T WAVE AMPLITUDE DECREASED                         1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  ELECTROCARDIOGRAM T WAVE INVERSION                                   2 (2.3%)          1 (1.4%)              1 (1.0%)           4 (1.3%)  
  HEART RATE INCREASED                                                 1 (1.2%)             0                      0              1 (0.3%)  
  HEART RATE IRREGULAR                                                 1 (1.2%)             0                      0              1 (0.3%)  
  NASAL MUCOSA BIOPSY                                                     0                 0                  1 (1.0%)           1 (0.3%)  
  NEUTROPHIL COUNT INCREASED                                              0                 0                  1 (1.0%)           1 (0.3%)  
  URINE ANALYSIS ABNORMAL                                                 0                 0                  1 (1.0%)           1 (0.3%)  
  WEIGHT DECREASED                                                        0                 0                  1 (1.0%)           1 (0.3%)  
  WHITE BLOOD CELL COUNT INCREASED                                        0                 0                  1 (1.0%)           1 (0.3%)  
METABOLISM AND NUTRITION DISORDERS                                                                                                          
  Total number of patients with at least one adverse event             6 (7.0%)          3 (4.2%)              1 (1.0%)          10 (3.3%)  
  Total number of events                                                  8                 5                      1                 14     
  DECREASED APPETITE                                                   1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  DEHYDRATION                                                          1 (1.2%)             0                      0              1 (0.3%)  
  DIABETES MELLITUS                                                    1 (1.2%)             0                      0              1 (0.3%)  
  FOOD CRAVING                                                         1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  HYPERCHOLESTEROLAEMIA                                                   0              1 (1.4%)                  0              1 (0.3%)  
  HYPONATRAEMIA                                                        1 (1.2%)             0                      0              1 (0.3%)  
  INCREASED APPETITE                                                   1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
MUSCULOSKELETAL AND CONNECTIVE TISSUE DISORDERS                                                                                             
  Total number of patients with at least one adverse event             5 (5.8%)         8 (11.1%)              7 (7.3%)          20 (6.5%)  
  Total number of events                                                  8                 11                    10                 29     
  ARTHRALGIA                                                           1 (1.2%)          1 (1.4%)              2 (2.1%)           4 (1.3%)  
  ARTHRITIS                                                            1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  BACK PAIN                                                            1 (1.2%)          3 (4.2%)              1 (1.0%)           5 (1.6%)  
  FLANK PAIN                                                              0              2 (2.8%)                  0              2 (0.7%)  
  MUSCLE SPASMS                                                           0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  MUSCULAR WEAKNESS                                                       0                 0                  1 (1.0%)           1 (0.3%)  
  MYALGIA                                                                 0              1 (1.4%)                  0              1 (0.3%)  
  PAIN IN EXTREMITY                                                    1 (1.2%)             0                      0              1 (0.3%)  
  SHOULDER PAIN                                                        1 (1.2%)             0                  2 (2.1%)           3 (1.0%)  
NEOPLASMS BENIGN, MALIGNANT AND UNSPECIFIED (INCL CYSTS AND POLYPS)                                                                         
  Total number of patients with at least one adverse event                0              1 (1.4%)              2 (2.1%)           3 (1.0%)  
  Total number of events                                                  0                 1                      3                 4      
  COLON CANCER                                                            0                 0                  1 (1.0%)           1 (0.3%)  
  MALIGNANT FIBROUS HISTIOCYTOMA                                          0                 0                  1 (1.0%)           1 (0.3%)  
  PROSTATE CANCER                                                         0              1 (1.4%)                  0              1 (0.3%)  
NERVOUS SYSTEM DISORDERS                                                                                                                    
  Total number of patients with at least one adverse event            12 (14.0%)        25 (34.7%)            22 (22.9%)         59 (19.3%) 
  Total number of events                                                  16                43                    42                101     
  AMNESIA                                                                 0              1 (1.4%)                  0              1 (0.3%)  
  BALANCE DISORDER                                                        0                 0                  1 (1.0%)           1 (0.3%)  
  BURNING SENSATION                                                       0              2 (2.8%)                  0              2 (0.7%)  
  COGNITIVE DISORDER                                                      0              1 (1.4%)                  0              1 (0.3%)  
  COMPLEX PARTIAL SEIZURES                                                0                 0                  1 (1.0%)           1 (0.3%)  
  COORDINATION ABNORMAL                                                   0                 0                  1 (1.0%)           1 (0.3%)  
  DIZZINESS                                                            2 (2.3%)         11 (15.3%)             9 (9.4%)          22 (7.2%)  
  HEADACHE                                                             7 (8.1%)          6 (8.3%)              3 (3.1%)          16 (5.2%)  
  HEMIANOPIA HOMONYMOUS                                                   0                 0                  1 (1.0%)           1 (0.3%)  
  HYPERSOMNIA                                                             0              1 (1.4%)                  0              1 (0.3%)  
  LETHARGY                                                                0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  PARAESTHESIA                                                            0              1 (1.4%)                  0              1 (0.3%)  
  PARAESTHESIA ORAL                                                       0                 0                  1 (1.0%)           1 (0.3%)  
  PARKINSON'S DISEASE                                                  1 (1.2%)             0                      0              1 (0.3%)  
  PAROSMIA                                                                0              1 (1.4%)                  0              1 (0.3%)  
  PARTIAL SEIZURES WITH SECONDARY GENERALISATION                          0              1 (1.4%)                  0              1 (0.3%)  
  PSYCHOMOTOR HYPERACTIVITY                                            1 (1.2%)             0                      0              1 (0.3%)  
  SOMNOLENCE                                                           2 (2.3%)          1 (1.4%)              3 (3.1%)           6 (2.0%)  
  STUPOR                                                                  0                 0                  1 (1.0%)           1 (0.3%)  
  SYNCOPE                                                                 0              2 (2.8%)              5 (5.2%)           7 (2.3%)  
  SYNCOPE VASOVAGAL                                                       0              1 (1.4%)                  0              1 (0.3%)  
  TRANSIENT ISCHAEMIC ATTACK                                              0              1 (1.4%)              2 (2.1%)           3 (1.0%)  
PSYCHIATRIC DISORDERS                                                                                                                       
  Total number of patients with at least one adverse event            10 (11.6%)        8 (11.1%)             11 (11.5%)         29 (9.5%)  
  Total number of events                                                  14                11                    15                 40     
  AGITATION                                                            2 (2.3%)             0                  3 (3.1%)           5 (1.6%)  
  ANXIETY                                                              1 (1.2%)             0                  3 (3.1%)           4 (1.3%)  
  COMPLETED SUICIDE                                                    1 (1.2%)             0                      0              1 (0.3%)  
  CONFUSIONAL STATE                                                    2 (2.3%)          1 (1.4%)              3 (3.1%)           6 (2.0%)  
  DELIRIUM                                                                0              1 (1.4%)                  0              1 (0.3%)  
  DELUSION                                                             1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  DEPRESSED MOOD                                                          0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  DISORIENTATION                                                       1 (1.2%)             0                      0              1 (0.3%)  
  HALLUCINATION                                                           0              1 (1.4%)                  0              1 (0.3%)  
  HALLUCINATION, VISUAL                                                   0              1 (1.4%)                  0              1 (0.3%)  
  INSOMNIA                                                             2 (2.3%)          2 (2.8%)                  0              4 (1.3%)  
  IRRITABILITY                                                         1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  LIBIDO DECREASED                                                        0              1 (1.4%)                  0              1 (0.3%)  
  LISTLESS                                                                0              1 (1.4%)                  0              1 (0.3%)  
  NIGHTMARE                                                               0              1 (1.4%)                  0              1 (0.3%)  
  RESTLESSNESS                                                            0                 0                  1 (1.0%)           1 (0.3%)  
RENAL AND URINARY DISORDERS                                                                                                                 
  Total number of patients with at least one adverse event             4 (4.7%)          3 (4.2%)              4 (4.2%)          11 (3.6%)  
  Total number of events                                                  5                 4                      4                 13     
  CALCULUS URETHRAL                                                       0              1 (1.4%)                  0              1 (0.3%)  
  DYSURIA                                                              1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  ENURESIS                                                                0                 0                  1 (1.0%)           1 (0.3%)  
  INCONTINENCE                                                            0                 0                  1 (1.0%)           1 (0.3%)  
  MICTURITION URGENCY                                                  1 (1.2%)          1 (1.4%)              1 (1.0%)           3 (1.0%)  
  NEPHROLITHIASIS                                                      1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  POLLAKIURIA                                                          1 (1.2%)             0                      0              1 (0.3%)  
REPRODUCTIVE SYSTEM AND BREAST DISORDERS                                                                                                    
  Total number of patients with at least one adverse event             2 (2.3%)          1 (1.4%)                  0              3 (1.0%)  
  Total number of events                                                  4                 1                      0                 5      
  BENIGN PROSTATIC HYPERPLASIA                                         1 (1.2%)          1 (1.4%)                  0              2 (0.7%)  
  PELVIC PAIN                                                          1 (1.2%)             0                      0              1 (0.3%)  
RESPIRATORY, THORACIC AND MEDIASTINAL DISORDERS                                                                                             
  Total number of patients with at least one adverse event            10 (11.6%)        10 (13.9%)            10 (10.4%)         30 (9.8%)  
  Total number of events                                                  15                22                    16                 53     
  ALLERGIC GRANULOMATOUS ANGIITIS                                         0              1 (1.4%)                  0              1 (0.3%)  
  COUGH                                                                3 (3.5%)          5 (6.9%)              6 (6.2%)          14 (4.6%)  
  DYSPHONIA                                                               0                 0                  1 (1.0%)           1 (0.3%)  
  DYSPNOEA                                                             1 (1.2%)          1 (1.4%)              1 (1.0%)           3 (1.0%)  
  EMPHYSEMA                                                            1 (1.2%)             0                      0              1 (0.3%)  
  EPISTAXIS                                                               0              2 (2.8%)              1 (1.0%)           3 (1.0%)  
  HAEMOPTYSIS                                                          1 (1.2%)             0                      0              1 (0.3%)  
  NASAL CONGESTION                                                     3 (3.5%)          3 (4.2%)              1 (1.0%)           7 (2.3%)  
  PHARYNGEAL ERYTHEMA                                                     0              1 (1.4%)                  0              1 (0.3%)  
  PHARYNGOLARYNGEAL PAIN                                                  0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  POSTNASAL DRIP                                                       1 (1.2%)             0                      0              1 (0.3%)  
  PRODUCTIVE COUGH                                                        0              1 (1.4%)                  0              1 (0.3%)  
  RALES                                                                1 (1.2%)             0                      0              1 (0.3%)  
  RESPIRATORY TRACT CONGESTION                                            0              1 (1.4%)                  0              1 (0.3%)  
  RHINORRHOEA                                                             0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
SKIN AND SUBCUTANEOUS TISSUE DISORDERS                                                                                                      
  Total number of patients with at least one adverse event            21 (24.4%)        42 (58.3%)            42 (43.8%)        105 (34.3%) 
  Total number of events                                                  47               111                    118               276     
  ACTINIC KERATOSIS                                                       0              1 (1.4%)                  0              1 (0.3%)  
  ALOPECIA                                                             1 (1.2%)             0                      0              1 (0.3%)  
  BLISTER                                                                 0              1 (1.4%)              5 (5.2%)           6 (2.0%)  
  COLD SWEAT                                                           1 (1.2%)             0                      0              1 (0.3%)  
  DERMATITIS ATOPIC                                                    1 (1.2%)             0                      0              1 (0.3%)  
  DERMATITIS CONTACT                                                      0                 0                  1 (1.0%)           1 (0.3%)  
  DRUG ERUPTION                                                        1 (1.2%)             0                      0              1 (0.3%)  
  ERYTHEMA                                                            9 (10.5%)         14 (19.4%)            15 (15.6%)         38 (12.4%) 
  HYPERHIDROSIS                                                        2 (2.3%)         8 (11.1%)              4 (4.2%)          14 (4.6%)  
  PRURITUS                                                             8 (9.3%)         26 (36.1%)            23 (24.0%)         57 (18.6%) 
  PRURITUS GENERALISED                                                    0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
  RASH                                                                 5 (5.8%)         11 (15.3%)            13 (13.5%)         29 (9.5%)  
  RASH ERYTHEMATOUS                                                       0                 0                  2 (2.1%)           2 (0.7%)  
  RASH MACULO-PAPULAR                                                     0              1 (1.4%)                  0              1 (0.3%)  
  RASH PAPULAR                                                            0              1 (1.4%)                  0              1 (0.3%)  
  RASH PRURITIC                                                           0              2 (2.8%)              1 (1.0%)           3 (1.0%)  
  SKIN EXFOLIATION                                                        0                 0                  1 (1.0%)           1 (0.3%)  
  SKIN IRRITATION                                                      3 (3.5%)          5 (6.9%)              6 (6.2%)          14 (4.6%)  
  SKIN ODOUR ABNORMAL                                                     0              1 (1.4%)                  0              1 (0.3%)  
  SKIN ULCER                                                           1 (1.2%)             0                      0              1 (0.3%)  
  URTICARIA                                                               0              1 (1.4%)              1 (1.0%)           2 (0.7%)  
SOCIAL CIRCUMSTANCES                                                                                                                        
  Total number of patients with at least one adverse event                0              1 (1.4%)                  0              1 (0.3%)  
  Total number of events                                                  0                 1                      0                 1      
  ALCOHOL USE                                                             0              1 (1.4%)                  0              1 (0.3%)  
SURGICAL AND MEDICAL PROCEDURES                                                                                                             
  Total number of patients with at least one adverse event             2 (2.3%)          2 (2.8%)              1 (1.0%)           5 (1.6%)  
  Total number of events                                                  2                 2                      1                 5      
  ACROCHORDON EXCISION                                                    0              1 (1.4%)                  0              1 (0.3%)  
  CATARACT OPERATION                                                   1 (1.2%)             0                  1 (1.0%)           2 (0.7%)  
  EYE LASER SURGERY                                                    1 (1.2%)             0                      0              1 (0.3%)  
  SKIN LESION EXCISION                                                    0              1 (1.4%)                  0              1 (0.3%)  
VASCULAR DISORDERS                                                                                                                          
  Total number of patients with at least one adverse event             3 (3.5%)          1 (1.4%)              4 (4.2%)           8 (2.6%)  
  Total number of events                                                  7                 1                      5                 13     
  HOT FLUSH                                                               0                 0                  1 (1.0%)           1 (0.3%)  
  HYPERTENSION                                                         1 (1.2%)             0                  2 (2.1%)           3 (1.0%)  
  HYPOTENSION                                                          2 (2.3%)             0                  1 (1.0%)           3 (1.0%)  
  ORTHOSTATIC HYPOTENSION                                              1 (1.2%)             0                      0              1 (0.3%)  
  WOUND HAEMORRHAGE                                                       0              1 (1.4%)                  0              1 (0.3%)  </code></pre>
</div>
</div>
</section></section><section id="shiny-module" class="level1"><h1>Shiny module</h1>
<p>ADaM dataset이 있는 경우, 이를 빠르게 분석할 수 있는 shiny module <a href="https://insightsengineering.github.io/teal/latest-tag/"><code>teal</code></a>을 소개한다. <br> teal을 이용, 직접 module을 수정/생성하여 분석을 진행할 수 있다.</p>
<p>아래는 <a href="https://pharmaverse.github.io/"><code>pharmaverse</code></a>에서 제공하는 <code>teal</code> 기반 Shiny App이다.<br>
Web 환경에서 직접 클릭하고, 필터링하고, 그래프를 확인해볼 수 있다: <a href="https://78vmhx-0-0.shinyapps.io/adam/">앱 바로 가기</a></p>
<hr>
<p>이 앱은 <code>ADSL</code>, <code>ADTTE</code> 등의 ADaM 데이터를 활용해 <strong>Demographic Table</strong>, <strong>Kaplan-Meier plot</strong>, <strong>Time to Event Table</strong> 등의 시각화를 제공한다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{lee2025,
  author = {LEE, Hojun},
  title = {ADaM {Compliant} {ADSL} {Dataset} {Generation}},
  date = {2025-04-08},
  url = {https://blog.zarathu.com/posts/2025-04-08-ADaM/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-lee2025" class="csl-entry quarto-appendix-citeas">
LEE, Hojun. 2025. <span>“ADaM Compliant ADSL Dataset Generation.”</span>
April 8, 2025. <a href="https://blog.zarathu.com/posts/2025-04-08-ADaM/">https://blog.zarathu.com/posts/2025-04-08-ADaM/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <guid>https://blog.zarathu.com/posts/2025-04-08-ADaM/</guid>
  <pubDate>Tue, 08 Apr 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-04-08-ADaM/img/logo.png" medium="image" type="image/png" height="164" width="144"/>
</item>
<item>
  <title>Comparison of Models for Competing Risk Analysis</title>
  <dc:creator>Suhyun Han</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-04-02-model_compare_index/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="theoretical-background" class="level1"><h1>[Theoretical background]</h1>
<p>Competing Risk 분석에서 모델의 예측 성능을 평가하기 위해 일반적으로 사용되는 지표는 다음과 같다.</p>
<section id="harrells-c-index" class="level3"><h3 class="anchored" data-anchor-id="harrells-c-index">1. Harrell’s C-index</h3>
<p><strong>개념</strong>:<br>
관찰된 사건 발생 순서와 모델의 예측 위험 순서가 얼마나 잘 일치하는지를 나타낸다.<br>
범위는 0부터 1이고, 0.5는 무작위 예측과 동일한 성능이며 1에 가까울수록 예측 성능이 좋다.</p>
<p><strong>계산 방법</strong>:<br>
Fine-Gray 모델을 이용하여 Competing Risk 데이터를 일반적인 생존 분석 형태로 변환한 후<br>
Cox 비례위험 모델로 분석한 결과를 바탕으로 계산한다.<br>
R에서는 <code><a href="https://rdrr.io/pkg/survival/man/finegray.html">survival::finegray()</a></code>로 데이터를 변환하고, <code><a href="https://rdrr.io/pkg/survival/man/coxph.html">survival::coxph()</a></code>와 <code><a href="https://rdrr.io/pkg/survival/man/concordance.html">survival::concordance()</a></code>로 C-index를 구한다.</p>
</section><section id="wolbers-c-index" class="level3"><h3 class="anchored" data-anchor-id="wolbers-c-index">2. Wolbers’ C-index</h3>
<p><strong>개념</strong>:<br>
Harrell’s C-index 개념을 확장하여 Competing Risk 상황에서 특정 원인의 사건 발생을 예측할 때 사용한다.</p>
<p><strong>계산 방법</strong>:<br>
Fine-Gray 모델(FGR)을 직접 적용하여 사건 특화된 C-index를 구한다.<br>
Bootstrapping을 이용하여 반복 표본추출로 신뢰구간을 얻는다.<br>
R에서는 <code>riskRegression::FGR()</code>과 <code>pec::cindex()</code>를 사용한다.</p>
</section><section id="auc-area-under-curve" class="level3"><h3 class="anchored" data-anchor-id="auc-area-under-curve">3. AUC (Area Under Curve)</h3>
<p><strong>개념</strong>:<br>
특정 시점에서 사건 발생 여부를 이진 분류로 간주하고, 예측의 민감도와 특이도를 종합적으로 평가한다.<br>
1에 가까울수록 예측 성능이 뛰어나고, 0.5에 가까울수록 무작위와 유사하다.</p>
<p><strong>계산 방법</strong>:<br><code>riskRegression::Score()</code> 함수를 이용해 특정 시점(예: 5년, 10년)의 AUC와 신뢰구간을 계산한다.</p>
</section><section id="brier-score" class="level3"><h3 class="anchored" data-anchor-id="brier-score">4. Brier Score</h3>
<p><strong>개념</strong>:<br>
예측 확률과 실제 관찰된 사건의 발생 여부 간의 차이를 측정하는 지표이다.<br>
낮을수록 성능이 우수하고, 0에 가까울수록 완벽한 예측이다.</p>
<p><strong>계산 방법</strong>:<br>
AUC와 같은 방식으로 <code>riskRegression::Score()</code> 함수로 특정 시점의 Brier Score와 신뢰구간을 구한다.</p>
</section></section><section id="preprocess" class="level1"><h1>[Preprocess]</h1>
<p>melanoma 데이터에서 필요한 변수(sex, age, thickness, ulcer)를 추출하고, status 변수는 censor=0인 Competing Risk 형태로 재구성한다. 시간을 연 단위로 변환하여 분석을 위한 데이터(melanoma_dt)를 준비한다.<br><strong>status</strong>: 1=melanoma 사망, 2=생존, 3=melanoma 외 사망<br><strong>status_competing</strong>: 0=생존, 1=melanoma 사망, 2=melanoma 외 사망<br></p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/message.html">suppressPackageStartupMessages</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://r-datatable.com">data.table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/therneau/survival">survival</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org">magrittr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/tagteam/riskRegression">riskRegression</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">prodlim</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pec</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>;<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://github.com/rstudio/rmarkdown">rmarkdown</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ulcer"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.SD</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">as.factor</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,.SDcols<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ulcer"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> </span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status_competing</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">==</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, levels<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> </span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time_Y</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">365.25</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"year"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"status"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"time"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"thickness"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ulcer"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sex"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"thickness"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ulcer"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Hist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time_Y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status_competing</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span>,</span>
<span>                           <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Hist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time_Y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status_competing</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">thickness</span>,</span>
<span>                           <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Hist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time_Y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status_competing</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ulcer</span>,</span>
<span>                           <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Hist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time_Y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status_competing</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">sex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">age</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">thickness</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ulcer</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
</section><section id="each-model" class="level1"><h1>[Each model]</h1>
<p>4가지 변수 조합의 모델을 설정하고 각각의 Harrell C-index, Wolbers C-index, AUC, Brier score를 계산한다. Wolbers’ C-index의 신뢰구간은 bootstrap 방법을 이용해 반복 표본추출로 계산한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/invisible.html">invisible</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/capture.output.html">capture.output</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span> </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Harrell_C_index_info</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_data_for_Harrell_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/finegray.html">finegray</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/formula.html">as.formula</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Surv(time_Y, status_competing) ~ "</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,collapse<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"+"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, etype<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Harrell_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/coxph.html">coxph</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/formula.html">as.formula</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Surv(fgstart, fgstop, fgstatus) ~ "</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,collapse<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"+"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_data_for_Harrell_C_index</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Harrell_C_index_info</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/concordance.html">concordance</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Harrell_C_index</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Wolbers_C_index_info</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pec</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>object<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index</span>, formula<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, confInt<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span>, verbose<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, size<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, replace<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index_boot</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info_boot</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pec</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>object<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index_boot</span>, formula<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, verbose<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info_boot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AppCindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FGR</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_lower</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/quantile.html">quantile</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span>, probs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_upper</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/quantile.html">quantile</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span>, probs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># AUC_and_Brier_info</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_AUC_and_Brier</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, formula<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> , data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, times<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, null.model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># return</span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Model"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">LETTERS</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,</span>
<span>              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Harrell_C_index"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Harrell_C_index_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">concordance</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"("</span>,</span>
<span>                                       <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Harrell_C_index_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">concordance</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">qnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.975</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">sqrt</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Harrell_C_index_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">var</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,</span>
<span>                                       <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Harrell_C_index_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">concordance</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">qnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.975</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">sqrt</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Harrell_C_index_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">var</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wolbers_C_index"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AppCindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"("</span>,</span>
<span>                                        <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_lower</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,</span>
<span>                                        <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_upper</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AUC_t=5"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" ("</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lower</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">upper</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"AUC_t=10"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" ("</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lower</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">upper</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Brier_t=5"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" ("</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lower</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">upper</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>              <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Brier_t=10"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" ("</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">lower</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,</span>
<span>                               <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.04f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier_info</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">upper</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> </span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
</section><section id="compare-two-model" class="level1"><h1>[Compare two model]</h1>
<section id="compare-harrell_c_index" class="level3"><h3 class="anchored" data-anchor-id="compare-harrell_c_index">1. Compare Harrell_C_index</h3>
<p>Harrell’s C-index는 각 모델 간의 concordance 함수를 통해 비교하며, 차이의 유의성을 Z 검정으로 평가한다</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Harrell_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_data_for_Harrell_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">survival</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/finegray.html">finegray</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/formula.html">as.formula</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Surv(time_Y, status_competing) ~ "</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,collapse<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"+"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, etype<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Harrell_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/coxph.html">coxph</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/formula.html">as.formula</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Surv(fgstart, fgstop, fgstatus) ~ "</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,collapse<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"+"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_data_for_Harrell_C_index</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span> </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_data_for_Harrell_C_index_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/finegray.html">finegray</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/formula.html">as.formula</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Surv(time_Y, status_competing) ~ "</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,collapse<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"+"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, etype<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Harrell_C_index_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/coxph.html">coxph</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/formula.html">as.formula</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste0</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Surv(fgstart, fgstop, fgstatus) ~ "</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/paste.html">paste</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>,collapse<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"+"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_data_for_Harrell_C_index_new</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ctest</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/survival/man/concordance.html">concordance</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Harrell_C_index</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Harrell_C_index_new</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">contr</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dtest</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">contr</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/matmult.html">%*%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/coef.html">coef</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ctest</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dvar</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">contr</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/matmult.html">%*%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/vcov.html">vcov</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ctest</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/matmult.html">%*%</a></span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">contr</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">z</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dtest</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">sqrt</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dvar</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value_temp</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/Normal.html">pnorm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/MathFun.html">abs</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">z</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> </span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value_temp</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.001</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;0.001"</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.03f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value_temp</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/function.html">return</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/unlist.html">unlist</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>Harrell_C_index<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/replace.html">replace</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Harrell_C_index</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Harrell_C_index"],"name":[1],"type":["chr"],"align":["left"]},{"label":["A"],"name":[2],"type":["chr"],"align":["left"]},{"label":["B"],"name":[3],"type":["chr"],"align":["left"]},{"label":["C"],"name":[4],"type":["chr"],"align":["left"]},{"label":["D"],"name":[5],"type":["chr"],"align":["left"]}],"data":[{"1":"A","2":"-","3":"0.001","4":"0.001","5":"<0.001"},{"1":"B","2":"0.001","3":"-","4":"0.656","5":"0.068"},{"1":"C","2":"0.001","3":"0.656","4":"-","5":"0.059"},{"1":"D","2":"<0.001","3":"0.068","4":"0.059","5":"-"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</div>
</section><section id="compare-wolbers_c_index" class="level3"><h3 class="anchored" data-anchor-id="compare-wolbers_c_index">2. Compare Wolbers_C_index</h3>
<p>Wolbers’ C-index는 bootstrap을 이용한 paired-sample t-test로 두 모델 간 차이를 평가한다. bootstrapping이므로 pair(model1, model2)에 대해 1번만 코드 실행한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/invisible.html">invisible</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/capture.output.html">capture.output</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>   <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Wolbers_C_index</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, size<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, replace<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index_boot</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info_boot</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pec</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>object<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index_boot</span>, formula<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, verbose<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info_boot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AppCindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FGR</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># A와 B,C,D 비교, B와 C,D 비교, C와 D 비교 </span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">20</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq_len</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, size<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/nrow.html">nrow</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, replace<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">indices_new</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index_boot_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data_new</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info_boot_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">pec</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>object<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_for_Wolbers_C_index_boot_new</span>, formula<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">boot_data_new</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, verbose<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>      <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values_new</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">b</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Wolbers_C_index_info_boot_new</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AppCindex</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">FGR</span></span>
<span>    <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span>      </span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value_temp</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/t.test.html">t.test</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cindex_values_new</span>, mu<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p.value</span></span>
<span>    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/function.html">return</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value_temp</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.001</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"&lt;0.001"</span>,<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.03f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p_value_temp</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/unlist.html">unlist</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as.data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">rbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, use.names<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>Wolbers_C_index<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/replace.html">replace</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Wolbers_C_index</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Wolbers_C_index"],"name":[1],"type":["chr"],"align":["left"]},{"label":["A"],"name":[2],"type":["chr"],"align":["left"]},{"label":["B"],"name":[3],"type":["chr"],"align":["left"]},{"label":["C"],"name":[4],"type":["chr"],"align":["left"]},{"label":["D"],"name":[5],"type":["chr"],"align":["left"]}],"data":[{"1":"A","2":"-","3":"<0.001","4":"<0.001","5":"<0.001"},{"1":"B","2":"-","3":"-","4":"0.189","5":"0.278"},{"1":"C","2":"-","3":"-","4":"-","5":"0.165"},{"1":"D","2":"-","3":"-","4":"-","5":"-"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</div>
</section><section id="compare-auc_and_brier" class="level3"><h3 class="anchored" data-anchor-id="compare-auc_and_brier">3. Compare AUC_and_Brier</h3>
<p>AUC와 Brier Score는 riskRegression::Score 함수를 이용하여 각 모델 간의 성능 차이를 비교한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/invisible.html">invisible</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/utils/capture.output.html">capture.output</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_and_Brier</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_AUC_and_Brier</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  </span>
<span>  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_AUC_and_Brier_new</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">FGR</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Hist_formula_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, cause<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    </span>
<span>    <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">riskRegression</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Score</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_AUC_and_Brier</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fgr_model_AUC_and_Brier_new</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,</span>
<span>                                           formula<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Hist</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">time_Y</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">status_competing</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, data<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">melanoma_dt</span>, times<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, null.model<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>    </span>
<span>    <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"V1"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.03f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">contrasts</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/sprintf.html">sprintf</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%.03f"</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">contrasts</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/character.html">as.character</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">j</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_t_5</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>AUC_t_5<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_t_5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_t_5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["AUC_t_5"],"name":[1],"type":["chr"],"align":["left"]},{"label":["A"],"name":[2],"type":["chr"],"align":["left"]},{"label":["B"],"name":[3],"type":["chr"],"align":["left"]},{"label":["C"],"name":[4],"type":["chr"],"align":["left"]},{"label":["D"],"name":[5],"type":["chr"],"align":["left"]}],"data":[{"1":"A","2":"-","3":"0.006","4":"0.003","5":"0.001"},{"1":"B","2":"0.006","3":"-","4":"0.518","5":"0.053"},{"1":"C","2":"0.003","3":"0.518","4":"-","5":"0.074"},{"1":"D","2":"0.001","3":"0.053","4":"0.074","5":"-"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_t_10</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>AUC_t_10<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_t_10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_t_10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["AUC_t_10"],"name":[1],"type":["chr"],"align":["left"]},{"label":["A"],"name":[2],"type":["chr"],"align":["left"]},{"label":["B"],"name":[3],"type":["chr"],"align":["left"]},{"label":["C"],"name":[4],"type":["chr"],"align":["left"]},{"label":["D"],"name":[5],"type":["chr"],"align":["left"]}],"data":[{"1":"A","2":"-","3":"0.607","4":"0.272","5":"0.663"},{"1":"B","2":"0.607","3":"-","4":"0.075","5":"0.189"},{"1":"C","2":"0.272","3":"0.075","4":"-","5":"0.143"},{"1":"D","2":"0.663","3":"0.189","4":"0.143","5":"-"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Brier_t_5</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>Brier_t_5<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Brier_t_5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Brier_t_5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Brier_t_5"],"name":[1],"type":["chr"],"align":["left"]},{"label":["A"],"name":[2],"type":["chr"],"align":["left"]},{"label":["B"],"name":[3],"type":["chr"],"align":["left"]},{"label":["C"],"name":[4],"type":["chr"],"align":["left"]},{"label":["D"],"name":[5],"type":["chr"],"align":["left"]}],"data":[{"1":"A","2":"-","3":"0.058","4":"0.023","5":"0.008"},{"1":"B","2":"0.058","3":"-","4":"0.582","5":"0.076"},{"1":"C","2":"0.023","3":"0.582","4":"-","5":"0.163"},{"1":"D","2":"0.008","3":"0.076","4":"0.163","5":"-"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Brier_t_10</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/lapply.html">lapply</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_AUC_and_Brier</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">k</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/do.call.html">do.call</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">rbind</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">setnames</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>Brier_t_10<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"A"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"B"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"C"</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"D"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%&gt;%</a></span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/replace.html">replace</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">.</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>,<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model_var_list</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Brier_t_10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span>,<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">i</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table_2_Brier_t_10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["Brier_t_10"],"name":[1],"type":["chr"],"align":["left"]},{"label":["A"],"name":[2],"type":["chr"],"align":["left"]},{"label":["B"],"name":[3],"type":["chr"],"align":["left"]},{"label":["C"],"name":[4],"type":["chr"],"align":["left"]},{"label":["D"],"name":[5],"type":["chr"],"align":["left"]}],"data":[{"1":"A","2":"-","3":"0.380","4":"0.187","5":"0.760"},{"1":"B","2":"0.380","3":"-","4":"0.012","5":"0.125"},{"1":"C","2":"0.187","3":"0.012","4":"-","5":"0.102"},{"1":"D","2":"0.760","3":"0.125","4":"0.102","5":"-"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
</div>
</div>
<p>각 지표는 성능이 좋을수록 C-index와 AUC는 높게, Brier Score는 낮게 나타난다.<br>
모델 간 비교 시 p-value가 작을수록 두 모델 간 성능의 유의한 차이가 있음을 의미한다.p-value가 0.05 미만이면 통계적으로 유의하게 두 모델의 성능이 다름을 나타낸다.<br>
이러한 과정을 통해 Competing Risk 모델의 성능을 객관적으로 평가하고 비교할 수 있으며, 실무적 활용도가 높은 모델을 선택하는 데 중요한 기준을 제공한다.</p>


</section></section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{han2025,
  author = {Han, Suhyun},
  title = {Comparison of {Models} for {Competing} {Risk} {Analysis}},
  date = {2025-04-07},
  url = {https://blog.zarathu.com/posts/2025-04-02-model_compare_index/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-han2025" class="csl-entry quarto-appendix-citeas">
Han, Suhyun. 2025. <span>“Comparison of Models for Competing Risk
Analysis.”</span> April 7, 2025. <a href="https://blog.zarathu.com/posts/2025-04-02-model_compare_index/">https://blog.zarathu.com/posts/2025-04-02-model_compare_index/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <guid>https://blog.zarathu.com/posts/2025-04-02-model_compare_index/</guid>
  <pubDate>Mon, 07 Apr 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Sankey Diagram 텍스트 삽입</title>
  <dc:creator>YeJi Kang</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-03-27-Sankey/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="i.-sankey-plot이란" class="level1"><h1>I. Sankey Plot이란?</h1>
<p>Sankey Plot은 데이터의 흐름이나 분포를 시각적으로 표현하는 그래프다.</p>
<p>선(링크)의 굵기로 양 또는 비율을 보여주며, 한 집단에서 다른 집단으로 얼마나 이동했는지를 한눈에 확인할 수 있게 도와 전체적인 구조와 흐름을 직관적으로 이해할 수 있도록 한다.</p>
<section id="텍스트를-추가하려고-했더니" class="level2"><h2 class="anchored" data-anchor-id="텍스트를-추가하려고-했더니">텍스트를 추가하려고 했더니…</h2>
<p>R에는 <code>sankeyNetwork()</code> 함수로 Sankey Plot을 생성할 수 있는 기능이 존재한다. 이 함수는 <code>networkD3</code> 패키지에서 제공되며, 비교적 간단한 코드로 노드와 링크 데이터를 연결해 Sankey Plot을 만들 수 있다는 장점이 있다 (특히 마우스 오버 시 강조되거나, 노드를 드래그할 수 있는 등의 동작은 시각적으로 유용하다).</p>
<p>하지만 이 함수만으로는 각 링크 위에 텍스트를 추가하거나, 노드 옆에 퍼센트 또는 샘플 수(n=) 같은 세부 정보를 표시하는 것이 어렵다. 그 이유는 Sankey Plot 내부 요소들의 위치(x, y 좌표)가 JavaScript에서 실시간으로 계산되기 때문이다. 따라서 R에서는 그 위치 정보를 직접 참조하거나 조작할 수가 없어 <code>sankeyNetwork()</code>를 이용한 기본적인 시각화 요소 외에는 커스터마이징 옵션이 제한적이다.</p>
<p>따라서 Sankey Plot의 시각적 완성도를 높이기 위해서는 결국 R 코드와 JavaScript를 함께 사용하는 접근이 요구된다.</p>
</section></section><section id="ii.-onrender-소개" class="level1"><h1>II. <code>onRender()</code> 소개</h1>
<p><code>onRender()</code>는 R에서 만든 시각화 객체에 JavaScript 코드를 직접 삽입할 수 있도록 도와주는 함수이다. 이 함수는 <code>htmlwidgets</code> 패키지에서 제공되며, <code>networkD3</code>, <code>plotly</code>, <code>leaflet</code> 등 다양한 시각화 패키지와 함께 사용된다.</p>
<p><code>onRender()</code>를 통해 JavaScript를 삽입하면 그래프 내부의 구성 요소들을 세부적으로 수정할 수 있다. 이 함수는 Sankey 그래프가 완성된 이후 실행되기 때문에, 그래프가 브라우저에 나타난 상태에서 DOM 요소를 선택해 조작하는 것을 가능하게 한다. 즉, <code>onRender()</code>는 R에서 만든 시각화 객체를 JavaScript 수준에서 마무리 커스터마이징 할 수 있게 해주는 도구이며, 시각화 결과를 한층 더 직관적이고 풍부하게 만들어주는 역할을 한다.</p>
</section><section id="iii.-기본-sankey-diagram-만들기" class="level1"><h1>III. 기본 Sankey Diagram 만들기</h1>
<p>아래의 ‘sankeyNetwork()’ 함수의 정의를 참고한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://christophergandrud.github.io/networkD3/">networkD3</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sankeyNetwork</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Links</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Nodes</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Source</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Target</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Value</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">NodeID</span>, NodeGroup <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">NodeID</span>,</span>
<span>  LinkGroup <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, units <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>,</span>
<span>  colourScale <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">JS</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"d3.scaleOrdinal(d3.schemeCategory20);"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, fontSize <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">7</span>,</span>
<span>  fontFamily <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, nodeWidth <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>, nodePadding <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span>, margin <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>,</span>
<span>  height <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, width <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NULL</span>, iterations <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">32</span>, sinksRight <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center collapsed" data-bs-toggle="collapse" data-bs-target=".callout-1-contents" aria-controls="callout-1" aria-expanded="false" aria-label="Toggle callout">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Note
</div>
<div class="callout-btn-toggle d-inline-block border-0 py-1 ps-1 pe-0 float-end"><i class="callout-toggle"></i></div>
</div>
<div id="callout-1" class="callout-1-contents callout-collapse collapse">
<div class="callout-body-container callout-body">
<p>Arguments</p>
<p><img src="https://blog.zarathu.com/posts/2025-03-27-Sankey/img/Sankey_arguments.png" class="img-fluid"></p>
</div>
</div>
</div>
<p>예시로 사용할 sankeydata.csv는 치료 방식 A,B,C에 대한 다양한 Response Type을 정리한 테이블이다.</p>
<div class="cell">
<div class="cell-output cell-output-stderr">
<pre><code>Rows: 12 Columns: 5
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): Treatment_Group, Response_Type
dbl (3): N, IDsource, IDtarget

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.</code></pre>
</div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<thead><tr class="header">
<th style="text-align: left;">Treatment_Group</th>
<th style="text-align: left;">Response_Type</th>
<th style="text-align: right;">N</th>
<th style="text-align: right;">IDsource</th>
<th style="text-align: right;">IDtarget</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">A</td>
<td style="text-align: left;">CR (Complete Response)</td>
<td style="text-align: right;">6</td>
<td style="text-align: right;">0</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: left;">A</td>
<td style="text-align: left;">PR (Partial Response)</td>
<td style="text-align: right;">18</td>
<td style="text-align: right;">0</td>
<td style="text-align: right;">4</td>
</tr>
<tr class="odd">
<td style="text-align: left;">A</td>
<td style="text-align: left;">SD (Stable Disease)</td>
<td style="text-align: right;">13</td>
<td style="text-align: right;">0</td>
<td style="text-align: right;">5</td>
</tr>
<tr class="even">
<td style="text-align: left;">A</td>
<td style="text-align: left;">PD (Progressive Disease)</td>
<td style="text-align: right;">3</td>
<td style="text-align: right;">0</td>
<td style="text-align: right;">6</td>
</tr>
<tr class="odd">
<td style="text-align: left;">B</td>
<td style="text-align: left;">CR (Complete Response)</td>
<td style="text-align: right;">10</td>
<td style="text-align: right;">1</td>
<td style="text-align: right;">3</td>
</tr>
<tr class="even">
<td style="text-align: left;">B</td>
<td style="text-align: left;">PR (Partial Response)</td>
<td style="text-align: right;">15</td>
<td style="text-align: right;">1</td>
<td style="text-align: right;">4</td>
</tr>
<tr class="odd">
<td style="text-align: left;">B</td>
<td style="text-align: left;">SD (Stable Disease)</td>
<td style="text-align: right;">7</td>
<td style="text-align: right;">1</td>
<td style="text-align: right;">5</td>
</tr>
<tr class="even">
<td style="text-align: left;">B</td>
<td style="text-align: left;">PD (Progressive Disease)</td>
<td style="text-align: right;">5</td>
<td style="text-align: right;">1</td>
<td style="text-align: right;">6</td>
</tr>
<tr class="odd">
<td style="text-align: left;">B</td>
<td style="text-align: left;">NR (No Response)</td>
<td style="text-align: right;">3</td>
<td style="text-align: right;">1</td>
<td style="text-align: right;">7</td>
</tr>
<tr class="even">
<td style="text-align: left;">C</td>
<td style="text-align: left;">SD (Stable Disease)</td>
<td style="text-align: right;">12</td>
<td style="text-align: right;">2</td>
<td style="text-align: right;">5</td>
</tr>
<tr class="odd">
<td style="text-align: left;">C</td>
<td style="text-align: left;">PD (Progressive Disease)</td>
<td style="text-align: right;">22</td>
<td style="text-align: right;">2</td>
<td style="text-align: right;">6</td>
</tr>
<tr class="even">
<td style="text-align: left;">C</td>
<td style="text-align: left;">NR (No Response)</td>
<td style="text-align: right;">6</td>
<td style="text-align: right;">2</td>
<td style="text-align: right;">7</td>
</tr>
</tbody>
</table>
</div>
</div>
<p>이 가상 데이터를 기반으로 Sankey Plot을 생성한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://r-datatable.com">data.table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mydata</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fread</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"sankeydata.csv"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 출발과 도착 노드의 고유값 추출</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">nodes</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>id <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/unique.html">unique</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mydata</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Treatment_Group</span>, <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mydata</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">Response_Type</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, stringsAsFactors <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 링크 색상에 사용할 색상 팔레트</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">colourScale</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  d3.scaleOrdinal()</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    .domain(["linkgrp"])</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    .range(["gainsboro"].concat(d3.schemeCategory20))</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span></span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Sankey Plot 생성</span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://christophergandrud.github.io/networkD3/">networkD3</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sankeyNetwork</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>Links <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mydata</span>, Nodes <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">nodes</span>, Source <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'IDsource'</span>,</span>
<span>                   Target <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'IDtarget'</span>, Value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'N'</span>, NodeID <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'id'</span>,</span>
<span>                   fontSize <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>, nodeWidth <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">40</span>, margin <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>left <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">200</span>, right <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">250</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, colourScale <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">colourScale</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p>이를 실행하면 다음과 같은 Sankey Plot을 출력할 수 있다. <img src="https://blog.zarathu.com/posts/2025-03-27-Sankey/img/sankeyplot1.png" class="img-fluid" alt="sankeyplot1"></p>
<p>하지만 우리가 원하는 것은 각 node와 flow가 총 값에서 어느정도의 비율을 차지하는지, 그리고 각 node 샘플의 값이다. 이제 JavaScript를 빌려 이 Sankey plot에 필요한 텍스트를 추가해보겠다.</p>
</section><section id="iv.-텍스트-추가하기" class="level1"><h1>IV. 텍스트 추가하기</h1>
<p><code>htmlwidgets</code> 패키지의 <code>onRender()</code>를 사용해 Sankey plot을 수정한다.</p>
<section id="각-node-옆에-비율-및-개수-표시" class="level2"><h2 class="anchored" data-anchor-id="각-node-옆에-비율-및-개수-표시">각 node 옆에 비율 및 개수 표시</h2>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">htmlwidgets</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/htmlwidgets/man/onRender.html">onRender</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  function(el, x) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    var sankey = this.sankey;</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"></span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    d3.select(el).selectAll(".node text")</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      .text(function(d) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        var perc = (d.value / 120) * 100;  // 총 샘플 수: 120</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        return d.name + " " + perc.toFixed(1) + "% (n=" + d.value + ")";</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      })</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      .attr("x", function(d) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        return (d.x === 0) ? -10 : x.options.nodeWidth + 10;</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      })</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      .attr("text-anchor", function(d) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        return (d.x === 0) ? "end" : "start";</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      });</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  }</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><code>htmlwidgets::onRender(p, '...')</code>는 p라는 Sankey 플롯 객체에 JavaScript 코드를 추가해주는 함수다. 이 코드는 Sankey plot이 브라우저에 렌더링된 이후 실행된다.</p>
<p>위 코드는 먼저 <code>var sankey = this.sankey</code>로 Sankey Plot의 내부 구조를 가져온 뒤, <code>.selectALL (".node text")</code>로 Sankey의 모든 노드 텍스트를 선택한다.</p>
<p>이어서 <code>function(el, x) { ... }</code>의 el은 Sankey 그래프가 들어있는 HTML의 요소이고, x는 sankeyNetwork 함수에서 설정된 옵션들(fontSize, nodeWidth 등)을 담고 있다. 텍스트를 추가하기 위해서 이 function 안에 있는 <code>.text</code> 함수를 가장 중심적으로 사용한다:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.text</span>(<span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(d) {</span>
<span id="cb5-2">        var perc <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> (d.value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">87</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>;  <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span><span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">/</span> 총 샘플 수<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">87</span></span>
<span id="cb5-3">        return d.name <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" "</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">perc.toFixed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"% (n="</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> d.value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span>;</span>
<span id="cb5-4">      }</span></code></pre></div></div>
</div>
<p>이 부분은 d.name(각 노드의 이름)에 전체 값 대비 해당 노드가 차지하는 비율(%)과 개수(n=)를 텍스트로 함께 붙이는 역할을 한다. 여기서 120이라는 숫자는 전체 합계(sum of N)를 의미하기 때문에, 사용하는 데이터의 전체 값에 따라 이 숫자는 다르게 삽입한다. 혹은, R에서 전체 합을 미리 계산해서 <code>onRender()</code>에 넘겨주는 방식을 사용해도 된다.</p>
<p>그 밑에 있는 <code>.attr("x", function(d) {...})</code>와 <code>.attr("text-anchor", function(d) {...})</code>는 위치 설정에 관련된 코드다. 노드가 왼쪽에 있을 경우(d.x === 0)에는 텍스트를 왼쪽 바깥에 정렬하고, 그렇지 않으면 오른쪽 바깥에 정렬하도록 설정한다. 텍스트가 노드와 겹치거나 너무 떨어져 보일 경우, -10 또는 <code>x.options.nodeWidth + 10</code> 값을 조정해서 위치를 맞춰주면 된다.</p>
</section><section id="각-flow의-확률-텍스트-추가" class="level2"><h2 class="anchored" data-anchor-id="각-flow의-확률-텍스트-추가">각 flow의 확률 텍스트 추가</h2>
<p>아래 코드는 Sankey plot의 각 링크 위에 해당 링크가 출발 노드 전체에서 차지하는 비율(%)을 텍스트로 표시해주는 역할을 한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">htmlwidgets</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/htmlwidgets/man/onRender.html">onRender</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">p</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  function(el) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    var sankey = this.sankey;</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    var nodeWidth = sankey.nodeWidth();</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    var links = sankey.links();</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"></span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    // 각 링크에 대해 비율 계산 및 텍스트 추가</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    links.forEach(function(d) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      var outflow = d3.sum(d.source.sourceLinks, function(l) {</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        return l.value;</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      });</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"></span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      var ratio = (d.value / outflow) * 100;</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"></span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      // 텍스트 위치 계산 (링크 중간)</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      var startX = d.source.x + nodeWidth;</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      var startY = d.source.y + d.sy + (d.dy / 2);</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;"></span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      // 텍스트 추가</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">      d3.select(el).select("svg g")</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        .append("text")</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        .attr("text-anchor", "middle")</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        .attr("alignment-baseline", "middle")</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        .attr("x", startX + 25)</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        .attr("y", startY)</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">        .text(ratio.toFixed(1) + "%");</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">    });</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">  }</span></span>
<span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
</div>
<p><code>var sankey = this.sankey</code>는 sankey 객체를 가져오는 부분이며, nodeWidth는 노드의 너비, links는 그래프에 존재하는 모든 링크를 가져온다.</p>
<p>위 코드의 핵심은 <code>links.forEach(function(d) {...})</code> 반복문이다. 각 링크에 대해 다음과 같은 작업이 이루어진다:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1">var outflow <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">d3.sum</span>(d.source.sourceLinks, <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(l) {</span>
<span id="cb7-2">  return l.value;</span>
<span id="cb7-3">})</span></code></pre></div></div>
</div>
<p>이 부분은 d.source(해당 링크가 출발하는 노드)에서 outflow(나가는 전체 흐름의 합)을 계산한다.</p>
<p>이어서 그 링크가 차지하는 비율을 아래와 같이 계산한다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1">var ratio <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> (d.value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> outflow) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>;</span></code></pre></div></div>
</div>
<p>예를 들어 10명이 이 링크를 통해 이동했고, 전체 outflow가 40이면, 비율은 25%가 된다.</p>
<p>텍스트를 표시할 위치도 직접 계산된다:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1">var startX <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> d.source.x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> nodeWidth;</span>
<span id="cb9-2">var startY <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> d.source.y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> d.sy <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> (d.dy <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>);</span></code></pre></div></div>
</div>
<p>startX은 출발 노드의 x좌표와 노드 너비의 합으로, 텍스트의 x 좌표 위치는 노드의 오른쪽 바깥이 된다. 비슷하게 startY은 출발 노드의 y좌표, 링크가 시작되는 위치, 그리고 링크 높이의 절반의 합이다. 따라서 텍스트의 위치는 링크 중간의 y좌표가 된다. 이 계산 과정으로 텍스트는 선의 중간쯤 되는 위치에 뜨게 된다.</p>
<p>마지막으로 텍스트를 실제로 추가한다:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb10" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb10-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">d3.select</span>(el)<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.select</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"svg g"</span>)</span>
<span id="cb10-2">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.append</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text"</span>)</span>
<span id="cb10-3">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.attr</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"text-anchor"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"middle"</span>)</span>
<span id="cb10-4">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.attr</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"alignment-baseline"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"middle"</span>)</span>
<span id="cb10-5">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.attr</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"x"</span>, startX <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">25</span>)      </span>
<span id="cb10-6">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.attr</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"y"</span>, startY)</span>
<span id="cb10-7">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">.text</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ratio.toFixed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"%"</span>);</span></code></pre></div></div>
</div>
<p>이 부분에서 숫자 포맷, 텍스트 위치(x, y), 텍스트 내용을 자유롭게 커스터마이징할 수 있다. <code>d3.select(el).select("svg g").append("text")</code>로 텍스트 요소를 생성하고, <code>.attr("text-anchor", "middle")</code>과 <code>.attr("alignment-baseline", "middle")</code>을 통해 텍스트가 x와 y좌표의 가운데 정렬로 위치하도록 설정한다. x 좌표는 앞서 계산한 startX에 25를 더해 노드에서 약간 떨어진 위치에 텍스트가 뜨게 하고, y 좌표는 startY로 그대로 둔다.</p>
<p>마지막으로 <code>.text(ratio.toFixed(1) + "%")</code>를 통해 소수점 첫째자리까지 계산된 비율을 문자열로 표시한다. 소수점 없이 정수로만 표시하고 싶다면 <code>ratio.toFixed(0)</code>으로 바꿀 수 있고, 텍스트에 단어를 붙이고 싶다면 “비율:” + ratio.toFixed(1) + “%”처럼 바꿀 수도 있다.</p>
<p>이와 같이 추가적으로 코드를 작성한 것을 실행하면 처음 만들었던 Sankey plot에 원하는 텍스트가 추가된 것을 확인할 수 있다: <img src="https://blog.zarathu.com/posts/2025-03-27-Sankey/img/sankeyplot2.png" class="img-fluid" alt="sankeyplot1"></p>
</section></section><section id="v.-shiny-app-소개" class="level1"><h1>V. Shiny App 소개</h1>
<p>이제 다소 복잡한 이 모든 과정을 자동화한 Shiny App을 소개한다. 이 앱은 웹 기반 인터페이스를 통해 데이터 입력부터 시각화, 색상 커스터마이징, 다운로드까지 전 과정을 하나의 환경에서 처리할 수 있도록 설계되어 있다.</p>
<iframe src="https://6zgo1c-yejikang63.shinyapps.io/Sankey/" width="100%" height="400px" frameborder="0">
</iframe>
<p><code>Edit Sankey Data</code>에서 사용자는 플롯의 source와 target, value 값을 테이블 형태로 입력하거나 수정할 수 있고, <code>Color Options</code>에서 연결된 노드나 흐름을 선택하여 원하는 색상을 지정할 수 있다. 또한 전체 값에 대한 상대적인 비중을 백분율로 시각화할 수 있는 퍼센트 표시 기능이 포함되어 있으며 이 기능은 하나의 단계로 구성된 경우에만 활성화된다.</p>
<p>단계는 필요에 따라 <code>Add Stage</code> 버튼으로 추가하거나 삭제할 수 있어 다양한 구조의 데이터를 유연하게 구성할 수 있다. 완성된 시각화 결과는 HTML 파일로 저장 가능하며 데이터 자체는 CSV 또는 XLSX 파일로도 다운로드할 수 있다.</p>
<p>이 기능들을 실행시켜 생성한 Sankey Diagram의 예시다:</p>
<p><img src="https://blog.zarathu.com/posts/2025-03-27-Sankey/img/sankey_editor2.png" class="img-fluid" alt="sankeyeditor"> 이와 같이 Saneky Diagram Editor App은 Sankey Plot 제작 과정을 더 간단하고 효율적으로 만들기 위한 도구로, 기능성과 편의성을 갖추고 있다.</p>
</section><section id="vi.-마치며" class="level1"><h1>VI. 마치며</h1>
<p>이번 포스트에서는 <code>sankeyNetwork()</code>로 생성한 Sankey Plot에 JavaScript를 활용해 링크마다 비율을 표시하고, 노드 옆에 샘플 수와 비율을 함께 표시하는 방법을 살펴보았다. 기본적으로 R에서는 sankeyNetwork로 그래프를 쉽게 만들 수는 있지만, 텍스트나 위치와 같은 디테일한 조정은 제공되지 않는다. 그래서 <code><a href="https://rdrr.io/pkg/htmlwidgets/man/onRender.html">htmlwidgets::onRender()</a></code>를 활용해 JavaScript 코드를 직접 삽입하는 방식이 필요했고, 이를 통해 시각적으로 훨씬 더 풍부한 Sankey Plot을 만들 수 있다.</p>
<p>앞서 소개한 Shiny App은 이 과정을 생략할 수 있는 도구로, 빠르게 커스터마이징 된 Sankey Plot을 편집하고 저장할 수 있기 때문에 편의성을 재공한다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{kang2025,
  author = {Kang, YeJi},
  title = {Sankey {Diagram} {텍스트} {삽입}},
  date = {2025-03-26},
  url = {https://blog.zarathu.com/posts/2025-03-27-Sankey/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-kang2025" class="csl-entry quarto-appendix-citeas">
Kang, YeJi. 2025. <span>“Sankey Diagram 텍스트 삽입.”</span> March 26,
2025. <a href="https://blog.zarathu.com/posts/2025-03-27-Sankey/">https://blog.zarathu.com/posts/2025-03-27-Sankey/</a>.
</div></div></section></div> ]]></description>
  <category>shiny</category>
  <guid>https://blog.zarathu.com/posts/2025-03-27-Sankey/</guid>
  <pubDate>Wed, 26 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-03-27-Sankey/img/image.png" medium="image" type="image/png" height="130" width="144"/>
</item>
<item>
  <title>Intraclass Correlation Coefficient 공부하기</title>
  <dc:creator>Sungho Choi</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/</link>
  <description><![CDATA[ 
<!-- Google tag (gtag.js) -->
<script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script>





<section id="icc는-무엇인가" class="level1">
<h1>ICC는 무엇인가</h1>
<p><a href="https://en.wikipedia.org/wiki/Intraclass_correlation">Intraclass Correlation Coefficient (ICC)</a>는 집단으로 구성된 단위들에 대해 수치 측정값이 주어졌을 때, 동일한 집단 내 단위들이 서로 얼마나 유사한지를 나타내는 기술 통계량이다. 주로 측정 신뢰도를 평가할 때 사용되며, 특히 동일한 대상에 대해 여러 관측자(또는 반복 측정)가 수행한 측정의 일관성을 평가하는 데 유용하다.</p>
<p>ICC는 일반적인 상관계수와 달리 <strong>짝지어진 관측값</strong>이 아니라, 그룹으로 구성된 데이터에 대해 계산된다는 점에서 차이가 있다.</p>
<section id="icc-vs.-cohens-kappa" class="level2">
<h2 class="anchored" data-anchor-id="icc-vs.-cohens-kappa">ICC vs.&nbsp;Cohen’s Kappa</h2>
<p>ICC와 Cohen’s Kappa는 모두 측정자 간 일치도를 측정하는 도구지만, 적용되는 데이터의 특성과 해석 방식에서 차이가 있다.</p>
<p><strong>자료의 유형</strong></p>
<ul>
<li>ICC는 연속형 변수에 사용됨</li>
<li>Cohen’s Kappa는 범주형 변수에 사용됨</li>
</ul>
<p>즉 ICC는 “얼마나 수치들이 서로 비슷한가?”를 보는 것이고, Cohen’s kappa는 “서로 같은 범주로 분류했는가?”를 보는 것이다.</p>
<p><strong>평가방식</strong></p>
<ul>
<li>ICC는 분산을 기반으로 측정값 간의 상관 관계(유사성)를 분석함.</li>
<li>Cohen’s Kappa는 기대 일치도를 고려하여, 관측된 일치도와 우연히 일어날 수 있는 일치도의 차이를 보정함.</li>
</ul>
<p>즉, ICC는 총 변동성 중에서 집단 내 일관성에 기인하는 부분의 비율을 나타내는 것이고, Cohen’s Kappa는 실제 일치가 단순히 우연에 의한 것이 아님을 보장해 주는 것이다.</p>
<p><strong>측정자 수</strong></p>
<ul>
<li>ICC는 둘 이상의 평가자에게 적용할 수 있음. 측정 모델에 따라 다양한 변형등이 존재함.</li>
<li>Cohen’s Kappa는 전통적으로 두 평가자에 대한 일치도 측정에 사용되며, 다수 평가자일 경우에는 Fleiss’ kappa 등을 사용함.</li>
</ul>
</section>
<section id="icc의-사용분야" class="level2">
<h2 class="anchored" data-anchor-id="icc의-사용분야">ICC의 사용분야</h2>
<ul>
<li>심리학, 의학, 교육학 등에서 테스트 재현성 또는 평가자 간 신뢰도 분석</li>
<li>머신러닝/데이터 분석에서는 데이터 라벨링 일관성 확인</li>
<li>반복 측정 설계에서 측정의 안정성 검증</li>
</ul>
</section>
</section>
<section id="초기의-icc" class="level1">
<h1>초기의 ICC</h1>
<p>초기의 ICC 연구는 <strong>짝을 이루는 측정값</strong>에 초점을 맞췄으며, 처음 제안된 ICC 통계량은 <strong>Pearson 상관계수</strong>를 수정한 형태였다.</p>
<section id="초기-icc의-정의" class="level2">
<h2 class="anchored" data-anchor-id="초기-icc의-정의">초기 ICC의 정의</h2>
<p>Ronald Fisher가 제안한 초기 ICC는 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ar%20=%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20%5Cfrac%7B(x_%7Bn,1%7D%20-%20%5Cbar%7Bx%7D)(x_%7Bn,2%7D%20-%20%5Cbar%7Bx%7D)%7D%7BNs%5E2%7D%0A"></p>
<ul>
<li>데이터셋은 <img src="https://latex.codecogs.com/png.latex?N">개의 짝으로 구성됨</li>
<li>각 개체 <img src="https://latex.codecogs.com/png.latex?n">에 대해 두 개의 측정값 <img src="https://latex.codecogs.com/png.latex?(x_%7Bn,1%7D%20,%20x_%7Bn,2%7D)">가 존재함</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cbar%7Bx%7D">는 전체 데이터의 평균 <img src="https://latex.codecogs.com/png.latex?%0A%5Cbar%7Bx%7D%20=%20%5Cfrac%7B1%7D%7B2N%7D%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,1%7D%20+%20x_%7Bn,2%7D)%0A"></li>
<li><img src="https://latex.codecogs.com/png.latex?s%5E2">은 전체 데이터의 분산 <img src="https://latex.codecogs.com/png.latex?%0As%5E2%20=%20%5Cfrac%7B1%7D%7B2N%7D%20%5Cleft%5C%7B%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,1%7D%20-%20%5Cbar%7Bx%7D)%5E2%20+%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,2%7D%20-%20%5Cbar%7Bx%7D)%5E2%20%5Cright%5C%7D%0A"></li>
</ul>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: center;">r 값 범위</th>
<th style="text-align: center;">신뢰도의 정도</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">r ≤ 0.00</td>
<td style="text-align: center;">Poor</td>
</tr>
<tr class="even">
<td style="text-align: center;">0.00 &lt; r ≤ 0.20</td>
<td style="text-align: center;">Slight</td>
</tr>
<tr class="odd">
<td style="text-align: center;">0.20 &lt; r ≤ 0.40</td>
<td style="text-align: center;">Fair</td>
</tr>
<tr class="even">
<td style="text-align: center;">0.40 &lt; r ≤ 0.60</td>
<td style="text-align: center;">Moderate</td>
</tr>
<tr class="odd">
<td style="text-align: center;">0.60 &lt; r ≤ 0.80</td>
<td style="text-align: center;">Substantial</td>
</tr>
<tr class="even">
<td style="text-align: center;">0.80 &lt; r ≤ 1.00</td>
<td style="text-align: center;">Almost Perfect</td>
</tr>
</tbody>
</table>
</section>
<section id="pearson-상관계수와-icc의-차이점" class="level2">
<h2 class="anchored" data-anchor-id="pearson-상관계수와-icc의-차이점">Pearson 상관계수와 ICC의 차이점</h2>
<p>Pearson 상관계수와 ICC는 모두 연속형 데이터를 비교할 때 사용하는 지표이지만, 그들이 측정하는 대상과 의미는 전혀 다르다.</p>
<p>먼저 Pearson 상관계수는 두 변수 간의 선형 관계(일관된 증가/감소)를 측정한다. 예를 들어, 키와 몸무게처럼, 한 변수가 커질수록 다른 변수도 함께 커지는 경향이 있는지를 파악할 때 유용하다. 이때 값은 -1에서 +1 사이이며, 1에 가까울수록 완벽한 양의 선형 관계를 의미한다. 하지만 이 지표는 값 자체의 차이는 고려하지 않는다.</p>
<p>반면 ICC는 동일한 대상을 여러 번 측정했을 때 그 값이 얼마나 비슷하게 나오는지를 측정하는 지표아다. 즉 서로 다른 평가자나 시간에 따라 측정값이 달라졌을 때, 그 변화가 개인의 특성 차이 때문인지, 아니면 측정자의 오차 때문인지를 분리할 수 있다. ICC 값은 일반적으로 0에서 1 사이에 있으며, 1에 가까울수록 일관되고 신뢰할 수 있는 측정이라고 해석할 수 있다.</p>
<p>중요한 차이는 해석의 포인트에 있다:</p>
<p>Pearson 상관계수는 측정값 간의 선형 관계만을 보기 때문에, 두 평가자가 항상 10점 차이를 주더라도 여전히 상관계수는 1이 될 수 있다.</p>
<p>하지만 ICC는 ’값 자체가 얼마나 유사한지’를 보며, 위와 같은 상황에서는 일치도가 낮다고 판단한다. 즉 Pearson은 “패턴의 일관성”, ICC는 “값의 일치도”를 본다고 이해할 수 있다.</p>
<p>또한, Pearson 상관계수는 서로 다른 단위를 가진 두 변수(예: 키와 체중)에도 적용 가능하지만, ICC는 동일한 측정 단위 내에서만 적절하게 해석할 수 있다.</p>
</section>
<section id="개-이상의-값을-가진-그룹에서의-icc" class="level2">
<h2 class="anchored" data-anchor-id="개-이상의-값을-가진-그룹에서의-icc">3개 이상의 값을 가진 그룹에서의 ICC</h2>
<p>데이터셋이 각 그룹당 3개의 측정값을 가지는 경우, ICC는 다음과 같이 확장된다. <img src="https://latex.codecogs.com/png.latex?%0Ar%20=%20%5Cfrac%7B1%7D%7B3N%20s%5E2%7D%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20%5Cleft%5C%7B%0A(x_%7Bn,1%7D%20-%20%5Cbar%7Bx%7D)(x_%7Bn,2%7D%20-%20%5Cbar%7Bx%7D)%20+%0A(x_%7Bn,1%7D%20-%20%5Cbar%7Bx%7D)(x_%7Bn,3%7D%20-%20%5Cbar%7Bx%7D)%20+%0A(x_%7Bn,2%7D%20-%20%5Cbar%7Bx%7D)(x_%7Bn,3%7D%20-%20%5Cbar%7Bx%7D)%0A%5Cright%5C%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cbar%7Bx%7D">는 전체 평균 <img src="https://latex.codecogs.com/png.latex?%0A%5Cbar%7Bx%7D%20=%20%5Cfrac%7B1%7D%7B3N%7D%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,1%7D%20+%20x_%7Bn,2%7D%20+%20x_%7Bn,3%7D)%0A"></li>
<li><img src="https://latex.codecogs.com/png.latex?s%5E2">는 전체 분산 <img src="https://latex.codecogs.com/png.latex?%0As%5E2%20=%20%5Cfrac%7B1%7D%7B3N%7D%20%5Cleft%5C%7B%0A%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,1%7D%20-%20%5Cbar%7Bx%7D)%5E2%20+%0A%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,2%7D%20-%20%5Cbar%7Bx%7D)%5E2%20+%0A%5Csum_%7Bn=1%7D%5E%7BN%7D%20(x_%7Bn,3%7D%20-%20%5Cbar%7Bx%7D)%5E2%0A%5Cright%5C%7D%0A"> 여기서, 그룹의 크기(K)가 커질수록, 계산 과정에서 고려해야 할 교차항의 수도 증가한다.</li>
</ul>
<p>위 공식을 일반화하면 다음과 같아진다. <img src="https://latex.codecogs.com/png.latex?%0Ar%20=%20%5Cfrac%7BK%7D%7BK-1%7D%20%5Ccdot%20%5Cfrac%7B1%7D%7BNs%5E2%7D%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20(%5Cbar%7Bx%7D_n%20-%20%5Cbar%7Bx%7D)%5E2%20-%20%5Cfrac%7B1%7D%7BK-1%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?K">는 그룹당 데이터 개수</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cbar%7Bx_n%7D">는 <img src="https://latex.codecogs.com/png.latex?n">번째 그룹의 평균</li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?K=3">을 대입하면 위 공식과 완벽히 같아진다. 위 공식에 따르면, ICC값은 항상 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B-1%7D%7BK-1%7D">이상의 값을 가진다는 것을 알 수 있다. 따라서 ICC는 항상 <img src="https://latex.codecogs.com/png.latex?-1%20%5Cleq%20r%20%5Cleq%201">의 범위 안에서 존재하지만, <strong>데이터 개수(</strong><img src="https://latex.codecogs.com/png.latex?K">)가 많아질 수록 음수로 나올 가능성이 줄어든다.</p>
<p>또한 충분히 큰 <img src="https://latex.codecogs.com/png.latex?K">에 대해서, 다음과 같이 근사할 수도 있다. <img src="https://latex.codecogs.com/png.latex?%0Ar%20=%20%5Cfrac%7BK%7D%7BK-1%7D%20%5Ccdot%20%5Cfrac%7B1%7D%7BNs%5E2%7D%20%5Csum_%7Bn=1%7D%5E%7BN%7D%20(%5Cbar%7Bx%7D_n%20-%20%5Cbar%7Bx%7D)%5E2%0A"></p>
</section>
<section id="icc의-해석-및-한계점" class="level2">
<h2 class="anchored" data-anchor-id="icc의-해석-및-한계점">ICC의 해석 및 한계점</h2>
<p>ICC는 총 분산 중에서 그룹 간 변동이 차지하는 비율로 해석할 수 있다.</p>
<p>이상적인 데이터에서는 ICC 값이 0~1 사이에 있어야 하지만, 실제 샘플 데이터에서는 음수 값이 나올 수도 있다. 이는 Ronald Fisher가 ICC를 편향되지 않은 추정량으로 설계했기 때문이다. 따라서 모집단에서 ICC가 0일 경우, 표본 데이터에서는 음수 값이 나올 수 있다.</p>
</section>
</section>
<section id="icc의-체계화" class="level1">
<h1>ICC의 체계화</h1>
<section id="shrout-fleiss-1979" class="level2">
<h2 class="anchored" data-anchor-id="shrout-fleiss-1979">Shrout &amp; Fleiss (1979)</h2>
<p>Shrout &amp; Fleiss는 초기의 ICC를 체계적으로 분류하였다. 3가지 모델 유형(Model 1, 2, 3), 2가지 측정 방식(1: single, k: 평균)으로 분류하여 총 6가지 유형의 ICC를 정의하였다.</p>
<p><strong>모델 종류</strong></p>
<ul>
<li>one-way random
<ul>
<li>피험자 간의 변동만을 고려하는 모형</li>
<li>피험자 간의 차이에 대한 평가지의 일치도를 평가할 때 사용함</li>
<li>평가자의 효과는 고려하지 않고, 단순히 피험자 간의 일관성을 평가할 때 사용함</li>
</ul></li>
<li>two-way random
<ul>
<li>피험자 간 변동뿐만이 아니라, 평가자 간의 변동도 고려하는 모형</li>
<li>동일한 피험자에 대해 평가한 결과가 얼마나 일치하는지, 평가자들 간의 평가 결과가 얼마나 일관성 있는 지를 평가할 때 사용함</li>
</ul></li>
<li>two-way mixed
<ul>
<li>피험자 간의 변동과 평가자 간의 변동을 고려하는 모형</li>
<li>특정 평가자들이 고정되어 있을 때, 피험자 간의 일치도를 평가하는 데 사용함</li>
</ul></li>
</ul>
<p><strong>측정방식</strong></p>
<ul>
<li>단일 측도(single)
<ul>
<li>평가자 간에 얼마나 차이가 있는지 확인</li>
<li>각 평가자에 의해 한 번의 측정이 일어난 경우</li>
</ul></li>
<li>평균 측도(average)
<ul>
<li>평균값과 얼마나 차이가 있는지 확인</li>
<li>각 평가자에 의해 여러 번 측정이 일어난 경우</li>
</ul></li>
</ul>
</section>
<section id="mcgraw-wong-1996" class="level2">
<h2 class="anchored" data-anchor-id="mcgraw-wong-1996">McGraw &amp; Wong (1996)</h2>
<p>McGraw &amp; Wong은 Shrout &amp; Fleiss의 체계를 확장하여 총 10가지 ICC 형태를 정의하였다. 다음과 같은 총 3가지의 분류 기준을 제시하였다.</p>
<p><strong>모델 종류</strong></p>
<ul>
<li><p>one-way random</p></li>
<li><p>two-way random</p></li>
<li><p>two-way mixed</p></li>
</ul>
<p><strong>측정방식</strong></p>
<ul>
<li><p>단일 측도(single)</p></li>
<li><p>평균 측도(average)</p></li>
</ul>
<p><strong>정의(Definition Agreement)</strong></p>
<ul>
<li>일치도(consistency)
<ul>
<li>상대적 순위/경향이 일치하는지를 의미</li>
<li>평가자 간의 체계적인 차이는 무시하고, 변동성만 분석</li>
<li>포함하는 오차 : 우연한 변동</li>
<li>사용 상황 : 평가자가 고정되어 있고, 상대적 순위가 중요한 경우</li>
</ul></li>
<li>절대합의도(absolute agreement)
<ul>
<li>두 평가자의 결과가 완전히 같은지를 의미</li>
<li>평가자 간의 체계적인 차이까지도 고려</li>
<li>포함하는 오차 : 우연한 변동, 평가자 간 편향</li>
<li>사용 상황 : 정량적 측정이 실제 절대값의 일치도를 요구하는 경우</li>
</ul></li>
</ul>
</section>
<section id="icc-분류" class="level2">
<h2 class="anchored" data-anchor-id="icc-분류">ICC 분류</h2>
<p>아래의 표는 Shrout &amp; Fleiss와 McGraw &amp; Wong의 기준에 따라 ICC를 정리한 것이다.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 38%">
<col style="width: 23%">
<col style="width: 37%">
</colgroup>
<thead>
<tr class="header">
<th style="text-align: center;">McGraw and Wong</th>
<th style="text-align: center;">Shrout and Fleiss</th>
<th style="text-align: center;">Formulas for Calculating ICC</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">One-way random effects, absolute agreement, single rater/measurment</td>
<td style="text-align: center;">ICC(1,1)</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_W%7D%7BMS_R%20+(k+1)MS_W%7D"></td>
</tr>
<tr class="even">
<td style="text-align: center;">Two-way random effects, consistency, single rater/measurment</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%20+(k-1)MS_E%7D"></td>
</tr>
<tr class="odd">
<td style="text-align: center;">Two-way random effects, absolute agreement, single rater/measurment</td>
<td style="text-align: center;">ICC(2,1)</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%20+(k-1)MS_E%20+%20%5Cfrac%7Bk%7D%7Bn%7D%20(MS_C%20-%20MS_E%20)%7D"></td>
</tr>
<tr class="even">
<td style="text-align: center;">Two-way mixed effects, consistency , single rater/measurment</td>
<td style="text-align: center;">ICC(3,1)</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%20+(k-1)MS_E%7D"></td>
</tr>
<tr class="odd">
<td style="text-align: center;">Two-way mixed effects, absolute agreement, single rater/measurment</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%20+(k-1)MS_E%20+%20%5Cfrac%7Bk%7D%7Bn%7D%20(MS_C%20-%20MS_E%20)%7D"></td>
</tr>
<tr class="even">
<td style="text-align: center;">One-way random effects, absolute agreement, multiple rater/measurment</td>
<td style="text-align: center;">ICC(1,k)</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_W%7D%7BMS_R%7D"></td>
</tr>
<tr class="odd">
<td style="text-align: center;">Two-way random effects, consistency, multiple rater/measurment</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%7D"></td>
</tr>
<tr class="even">
<td style="text-align: center;">Two-way random effects, absolute agreement, multiple rater/measurment</td>
<td style="text-align: center;">ICC(2,k)</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%20+%20%5Cfrac%7BMS_C%20-MS_E%7D%7Bn%7D%7D"></td>
</tr>
<tr class="odd">
<td style="text-align: center;">Two-way mixed effects, consistency, multiple rater/measurment</td>
<td style="text-align: center;">ICC(3,k)</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%7D"></td>
</tr>
<tr class="even">
<td style="text-align: center;">Two-way mixed effects, absolute agreement, multiple rater/measurment</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;"><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BMS_R%20-%20MS_E%7D%7BMS_R%20+%20%5Cfrac%7BMS_C%20-MS_E%7D%7Bn%7D%7D"></td>
</tr>
</tbody>
</table>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?k"> : 평가자 수</li>
<li><img src="https://latex.codecogs.com/png.latex?n"> : 피험자 수</li>
<li><img src="https://latex.codecogs.com/png.latex?MS_R"> : mean square for rows</li>
<li><img src="https://latex.codecogs.com/png.latex?MS_W"> : mean square for residual sources of variance</li>
<li><img src="https://latex.codecogs.com/png.latex?MS_E"> : mean square for error</li>
<li><img src="https://latex.codecogs.com/png.latex?MS_C"> : mean square for columns</li>
</ul>
</section>
<section id="one-way-모델에서-consistency가-정의-되지-않는-이유" class="level2">
<h2 class="anchored" data-anchor-id="one-way-모델에서-consistency가-정의-되지-않는-이유">One-way 모델에서 Consistency가 정의 되지 않는 이유</h2>
<p>One-way random model은 평가자 효과를 모델에 포함하지 않기 때문이다. One-way 모델에서는 오직 피험자 간 차이만 고려하기 때문에 평가자 간 차이가 분산 구조에서 빠져있다. 따라서 평가자 간 일관성을 측정할 수 없다. 결과적으로 One-way 모델은 “Agreement”는 가능하지만 “Consistency”는 정의할 수 없다.</p>
</section>
<section id="icc-방식을-정하는-법" class="level2">
<h2 class="anchored" data-anchor-id="icc-방식을-정하는-법">ICC 방식을 정하는 법</h2>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/img/icc.figure.png" class="img-fluid figure-img"></p>
<figcaption>ICC 방식을 정하는 과정</figcaption>
</figure>
</div>
<p>위 그림은 ICC 유형을 선택하는 의사결정 흐름도를 시각적으로 잘 정리한 자료이다. ICC는 McGraw &amp; Wong이 제안한 모델을 기준으로 총 10가지 유형이 존재한다. 각각의 선택은 연구 설계에 따라 달라지며, 다음과 같은 과정을 거치면 적절한 유형을 결정할 수 있다.</p>
<p>첫 번째로, 연구 유형이 Test-Retest / Intra-rater Reliability인지, Inter-rater Reliability인지 알아본다. 만일 Test-Retest / Intra-rater Reliability라면, 같은 평가자가 같은 대상을 반복 측정하는 경우이기 때문에 Two-way Mixed Effects 모델을 사용한다. 반면에 Inter-rater Reliability라면 여러 평가자가 각 대상을 평가하는 경우이기 때문에, 다음 과정으로 넘어간다.</p>
<p>연구 유형이 Inter-rater Reliability일 때 모든 피평가자가 같은 평가자에게 평가되지 않았다면, 일부 평가자만 일부 피평가자를 평가하므로 One-way Random Effects 모델을 사용한다. 만일 모든 피평가자가 같은 평가자에게 평가되었다면, 평가자가 무작위 집단이라면 Two-way Random Effects 모델을 사용하고, 특정 평가자에 한정된다면 Two-way Mixed Effects 모델을 사용한다. 위 과정을 통해 ICC의 모델 중 어떤 것을 사용할 지 선택할 수 있다.</p>
<p>두 번째로 측정 프로토콜이 어떤 지 확인한다. 만일 한 번만 측정한다면 single rater/measurment을 사용하고, 여러 번 측정 후 평균을 사용한다면 multiple rater/measurment을 사용한다.</p>
<p>마지막으로 연구에서 중요한 것이 절대적 일치인지, 경향성 일치인지 판단한다. 진단 일치나 채점 점수 등 수치 자체가 같아야 하는 경우라면 Absolute Agreement를 사용하고, 순위 유지가 중요한 경우라면 Consistency를 사용하면 된다.</p>
</section>
<section id="대중적으로-사용되는-icc-방식" class="level2">
<h2 class="anchored" data-anchor-id="대중적으로-사용되는-icc-방식">대중적으로 사용되는 ICC 방식</h2>
<p>연구 현장에서 많이 쓰이는 ICC는 다음과 같다.</p>
<ul>
<li>ICC(2,1): Two-way random effects, single rater/measurment, absolute agreement</li>
<li>ICC(2,k): Two-way random effects, multiple rater/measurment, absolute agreement</li>
</ul>
<p>Two-way random effects 모델은 평가자와 피평가자 모두 무작위 샘플링된 것으로 간주한다. 따라서 평가자도 연구의 일반적인 모집단에서 랜덤하게 선택된 경우를 반영할 수 있어 일반화 가능성이 높다.</p>
<p>Absolute agreement는 단순히 측정값 간의 일관성만이 아니라, 측정값 자체가 정확히 일치하는지를 따진다. 이는 consistency보다 더 보수적이고 엄격한 방식이다. 또한 consistency는 평가자의 bias를 왜곡하는 반면, Absolute agreement는 systematic bias까지 감지할 수 있다.</p>
<p>따라서 Two-way random effects, Absolute agreement를 사용하는 ICC(2,1), ICC(2,k)를 실제 연구에서 많이 사용한다.</p>
</section>
</section>
<section id="모던-icc" class="level1">
<h1>모던 ICC</h1>
<p>초기 ICC는 ANOVA(분산 분석) 기반의 접근 방식으로 시작되었으나, 이후 <strong>랜덤 효과 모형(Random Effects Model)</strong>을 통해 발전하였다.</p>
<section id="랜덤-효과-모형에서의-icc" class="level2">
<h2 class="anchored" data-anchor-id="랜덤-효과-모형에서의-icc">랜덤 효과 모형에서의 ICC</h2>
<p>모던 ICC는 다음과 같은 one-way random effects 모형에서 정의된다. <img src="https://latex.codecogs.com/png.latex?%0AY_%7Bij%7D%20=%20%5Cmu%20+%20%5Calpha_j%20+%20%5Cvarepsilon_%7Bij%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?Y_%7Bij%7D">는 <img src="https://latex.codecogs.com/png.latex?j">번째 그룹의 <img src="https://latex.codecogs.com/png.latex?i">번째 측정값</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cmu">는 모집단 전체의 평균</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Calpha_j">는 그룹 <img src="https://latex.codecogs.com/png.latex?j">에 해당하는 랜덤효과</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cvarepsilon_%7Bij%7D">는 오차항</li>
</ul>
</section>
<section id="모던-icc의-공식" class="level2">
<h2 class="anchored" data-anchor-id="모던-icc의-공식">모던 ICC의 공식</h2>
<p>모던 ICC는 다음과 같이 정의된다. <img src="https://latex.codecogs.com/png.latex?%0AICC%20=%20%5Cfrac%7B%5Csigma%5E2_%7B%5Calpha%7D%7D%7B%5Csigma%5E2_%7B%5Calpha%7D%20+%20%5Csigma%5E2_%7B%5Cvarepsilon%7D%7D%0A"></p>
<p>분자<img src="https://latex.codecogs.com/png.latex?(%5Csigma%5E2_%7B%5Calpha%7D)">는 그룹간 분산을 의미하고, 분모<img src="https://latex.codecogs.com/png.latex?(%5Csigma%5E2_%7B%5Calpha%7D+%5Csigma%5E2_%7B%5Cvarepsilon%7D)">은 전체 분산을 의미한다.</p>
<p>즉 ICC는 전체 변동에서 그룹간 변동이 차지하는 비율이 된다. 따라서 ICC값이 클 수록 같은 그룹 내에서 값들이 더 유사하다는 것을 알 수 있다.</p>
<p><strong>증명</strong></p>
<p><img src="https://latex.codecogs.com/png.latex?Y_%7Bij%7D%20=%20%5Cmu%20+%20%5Calpha_j%20+%20%5Cvarepsilon_%7Bij%7D">에서 <img src="https://latex.codecogs.com/png.latex?%5Calpha_i%20%5Csim%20N(0,%20%5Csigma%5E2_%7B%5Calpha%7D)">, <img src="https://latex.codecogs.com/png.latex?%5Cepsilon_%7Bij%7D%20%5Csim%20N(0,%20%5Csigma%5E2_%7B%5Cvarepsilon%7D)">이고 <img src="https://latex.codecogs.com/png.latex?%5Calpha_i">와 <img src="https://latex.codecogs.com/png.latex?%5Cvarepsilon_%7Bij%7D">은 서로 독립이다.</p>
<p><img src="https://latex.codecogs.com/png.latex?Var(Y_%7Bij%7D)%20=%20%5Csigma%5E2_%7B%5Cvarepsilon%7D%20+%20%5Csigma%5E2_%7B%5Calpha%5E2%7D"></p>
<img src="https://latex.codecogs.com/png.latex?%5Cbegin%7Balign%7D%0ACov(Y_%7Bij%7D,%20Y_%7Bik%7D)%20&amp;=%20Cov(%5Cmu%20+%20%5Calpha_%7Bi%7D%20+%20%5Cvarepsilon_%7Bij%7D,%20%5Cmu%20+%20%5Calpha_%7Bi%7D%20+%20%5Cvarepsilon_%7Bik%7D)%5C%5C%0A%20%20&amp;=%20Cov(%5Calpha_i%20+%20%5Cvarepsilon_%7Bij%7D,%5Calpha_i%20+%20%5Cvarepsilon_%7Bik%7D)%20%5C%5C%0A%20%20&amp;=%20Cov(%5Calpha_i%20,%20%5Calpha_i)%20+%20Cov(%5Calpha_i,%20%5Cvarepsilon_%7Bik%7D)%20+%20Cov(%5Cvarepsilon_%7Bij%7D,%20%5Calpha_i)%20+%20Cov(%5Cvarepsilon_%7Bij%7D,%20%5Cvarepsilon_%7Bik%7D)%20%5C%5C%0A%20%20&amp;=%20Var(%5Calpha_i)%20=%20%5Csigma%5E2_%5Calpha%0A%5Cend%7Balign%7D">
</section>
<section id="모던-icc의-장점" class="level2">
<h2 class="anchored" data-anchor-id="모던-icc의-장점">모던 ICC의 장점</h2>
<ul>
<li>항상 0 이상이다
<ul>
<li>초기 ICC는 표본에서 음수값이 나올 수 있었음</li>
</ul></li>
<li>ANOVA와 결합 가능하다 <img src="https://latex.codecogs.com/png.latex?%5Crightarrow"> 샘플 개수가 달라도 적용 가능하다
<ul>
<li>초기 ICC는 같은 크기의 그룹을 가정함</li>
<li>하지만 ANOVA 기반 ICC는 데이터 개수가 달라도 계산 가능</li>
</ul></li>
<li>공변량을 포함할 수 있다
<ul>
<li>공변량을 통제한 후에도 같은 그룹 내에서 얼마나 유사한지 평가할 수 있음</li>
</ul></li>
<li>복잡한 데이터 설계에 유리하다</li>
</ul>
</section>
<section id="모던-icc의-한계점" class="level2">
<h2 class="anchored" data-anchor-id="모던-icc의-한계점">모던 ICC의 한계점</h2>
<ul>
<li>샘플 ICC가 실제 모집단 ICC보다 클 가능성이 높다
<ul>
<li>초기 ICC는 편향되지 않은 추정량임</li>
<li>모던 ICC는 항상 0 이상이므로, 모집단의 ICC가 정확히 0일 때에도 샘플에서 ICC가 0보다 크게 나올 가능성이 있음</li>
<li>즉, 양의 편향을 가짐</li>
</ul></li>
<li>여러 종류의 ICC가 존재<img src="https://latex.codecogs.com/png.latex?%5Crightarrow">어떤 ICC를 사용할지 논란이 된다
<ul>
<li>연구자마다 다른 ICC 통계량을 사용하며, 각 방법이 서로 다른 결과를 낼 수 있음</li>
<li>특정 연구 목적에 적합한 ICC를 신중하게 선택해야 함</li>
</ul></li>
</ul>
</section>
</section>
<section id="icc-r-실습" class="level1">
<h1>ICC R 실습</h1>
<section id="특정-icc-방법을-실행하는-r코드" class="level2">
<h2 class="anchored" data-anchor-id="특정-icc-방법을-실행하는-r코드">특정 ICC 방법을 실행하는 R코드</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(irr)</span>
<span id="cb1-2"></span>
<span id="cb1-3">ratings <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb1-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Rater1 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>),</span>
<span id="cb1-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Rater2 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb1-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Rater3 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb1-7">)</span>
<span id="cb1-8"></span>
<span id="cb1-9">result <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">icc</span>(ratings, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">model =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"twoway"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">type =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"agreement"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">unit =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"single"</span>)</span>
<span id="cb1-10"></span>
<span id="cb1-11"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">print</span>(result)</span></code></pre></div></div>
<p><strong>출력</strong></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1">Single Score Intraclass Correlation</span>
<span id="cb2-2"></span>
<span id="cb2-3">   Model<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> twoway </span>
<span id="cb2-4">   Type <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> agreement </span>
<span id="cb2-5"></span>
<span id="cb2-6">   Subjects <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span> </span>
<span id="cb2-7">     Raters <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span> </span>
<span id="cb2-8">   <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ICC</span>(A,<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>) <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.792</span></span>
<span id="cb2-9"></span>
<span id="cb2-10"> F<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>Test, H0<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> r0 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> ; H1<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> r0 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span> </span>
<span id="cb2-11">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">F</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>,<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">9.09</span>) <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">15.1</span> , p <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.000482</span> </span>
<span id="cb2-12"></span>
<span id="cb2-13"> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">95</span>%<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span>Confidence Interval <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> ICC Population Values<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span></span>
<span id="cb2-14">  <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.375</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> ICC <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.973</span></span></code></pre></div></div>
</section>
<section id="shrout-fleiss의-6개-방법을-모두-보여주는-r코드" class="level2">
<h2 class="anchored" data-anchor-id="shrout-fleiss의-6개-방법을-모두-보여주는-r코드">Shrout &amp; Fleiss의 6개 방법을 모두 보여주는 R코드</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(psych)</span>
<span id="cb3-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ICC</span>(ratings)</span></code></pre></div></div>
<p><strong>출력</strong></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">Call<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ICC</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x =</span> ratings)</span>
<span id="cb4-2"></span>
<span id="cb4-3">Intraclass correlation coefficients </span>
<span id="cb4-4">                         type  ICC  F df1 df2       p lower bound upper bound</span>
<span id="cb4-5">Single_raters_absolute   ICC1 <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.79</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>  <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.00072</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.37</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.97</span></span>
<span id="cb4-6">Single_random_raters     ICC2 <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.79</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.00085</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.37</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.97</span></span>
<span id="cb4-7">Single_fixed_raters      ICC3 <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.82</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.00085</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.40</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.98</span></span>
<span id="cb4-8">Average_raters_absolute ICC1k <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.92</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">12</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>  <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.00072</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.64</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.99</span></span>
<span id="cb4-9">Average_random_raters   ICC2k <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.92</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.00085</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.64</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.99</span></span>
<span id="cb4-10">Average_fixed_raters    ICC3k <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.93</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>   <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.00085</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.66</span>        <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.99</span></span>
<span id="cb4-11"></span>
<span id="cb4-12"> Number of subjects <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>     Number of Judges <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span>  <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span></span>
<span id="cb4-13">See the help file <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> a discussion of the other <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span> McGraw and Wong estimates,</span></code></pre></div></div>
</section>
<section id="lmer-방식의-모던-icc-r코드" class="level2">
<h2 class="anchored" data-anchor-id="lmer-방식의-모던-icc-r코드">lmer 방식의 모던 ICC R코드</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb5-2">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">subject =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rep</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">each =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb5-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rep</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">times =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>),</span>
<span id="cb5-4">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">score =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">80</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">82</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">81</span>,</span>
<span id="cb5-5">            <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">75</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">76</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">74</span>,</span>
<span id="cb5-6">            <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">90</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">89</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">91</span>,</span>
<span id="cb5-7">            <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">70</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">72</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">71</span>,</span>
<span id="cb5-8">            <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">85</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">86</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">84</span>)</span>
<span id="cb5-9">)</span>
<span id="cb5-10"></span>
<span id="cb5-11"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(lme4)</span>
<span id="cb5-12"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(performance)</span>
<span id="cb5-13"></span>
<span id="cb5-14">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lmer</span>(score <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> (<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> subject) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> (<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> rater), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> df)</span>
<span id="cb5-15"></span>
<span id="cb5-16"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">icc</span>(model)</span></code></pre></div></div>
<p><strong>출력</strong></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Intraclass Correlation Coefficient</span></span>
<span id="cb6-2"></span>
<span id="cb6-3">    Adjusted ICC<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.985</span></span>
<span id="cb6-4">  Unadjusted ICC<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.985</span></span></code></pre></div></div>
</section>
<section id="모던-icc에서-ci-구하기" class="level2">
<h2 class="anchored" data-anchor-id="모던-icc에서-ci-구하기">모던 ICC에서 CI 구하기</h2>
<p>lmer을 이용한 모던 ICC에서는 CI를 직접 제공하지 않는 것을 알 수 있다. 이는 ICC는 비선형 함수이고, <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">자체는 정규분포를 따르지 않기 때문에 정규 근사(CI ≈ estimate ± 1.96 × SE)가 잘 맞지 않기 때문이다.</p>
<p>그러나 bootstrapping 방식을 통해서 충분히 CI를 추정할 수 있다. bootstrap은 데이터를 반복적으로 다시 샘플링해서, 어떤 통계량의 표본 분포를 직접 만들어내는 기법이다. 원래 데이터를 기준으로 같은 크기의 샘플을 복원추출로 다시 만든 후, 새롭게 만들어진 데이터로 lmer을 적합한다. 그 모델에서 ICC를 계산하고, 이 과정을 계속 반복하여 ICC의 분포를 만든다. 그 값들의 분포에서 CI을 추출할 수 있다.</p>
<p>아래는 그 과정을 실행하는 코드이다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(lme4)</span>
<span id="cb7-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(performance)</span>
<span id="cb7-3"></span>
<span id="cb7-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span>)</span>
<span id="cb7-5"></span>
<span id="cb7-6">n_subjects <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">30</span></span>
<span id="cb7-7">n_raters <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span></span>
<span id="cb7-8">ratings_per_subject <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span></span>
<span id="cb7-9"></span>
<span id="cb7-10">subjects <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span>n_subjects</span>
<span id="cb7-11">raters <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span>n_raters</span>
<span id="cb7-12"></span>
<span id="cb7-13"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 데이터프레임 생성</span></span>
<span id="cb7-14">df <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">do.call</span>(rbind, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lapply</span>(subjects, <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(s) {</span>
<span id="cb7-15">  r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(raters, ratings_per_subject)</span>
<span id="cb7-16">  mu <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mean =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">75</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sd =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>)  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># subject 고유의 평균 실력</span></span>
<span id="cb7-17">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb7-18">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">subject =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">factor</span>(s),</span>
<span id="cb7-19">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">factor</span>(r),</span>
<span id="cb7-20">    <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">score =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(ratings_per_subject, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mean =</span> mu, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">sd =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb7-21">  )</span>
<span id="cb7-22">}))</span>
<span id="cb7-23"></span>
<span id="cb7-24"></span>
<span id="cb7-25">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lmer</span>(score <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> (<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> subject) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> (<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">|</span> rater), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> df, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">REML =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb7-26"></span>
<span id="cb7-27">performance<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">icc</span>(model, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ci =</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">iterations =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1000</span>)</span></code></pre></div></div>
<p><strong>출력</strong></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Intraclass Correlation Coefficient</span></span>
<span id="cb8-2"></span>
<span id="cb8-3">    Adjusted ICC<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.794</span> [<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.472</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.934</span>]</span>
<span id="cb8-4">  Unadjusted ICC<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.794</span> [<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.472</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.934</span>]</span></code></pre></div></div>


</section>
</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{choi2025,
  author = {Choi, Sungho},
  title = {Intraclass {Correlation} {Coefficient} {공부하기}},
  date = {2025-03-24},
  url = {https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-choi2025" class="csl-entry quarto-appendix-citeas">
Choi, Sungho. 2025. <span>“Intraclass Correlation Coefficient
공부하기.”</span> March 24, 2025. <a href="https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/">https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/</a>.
</div></div></section></div> ]]></description>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/</guid>
  <pubDate>Mon, 24 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-03-24-IntraclassCorrelationCoefficient/img/icc.png" medium="image" type="image/png" height="108" width="144"/>
</item>
<item>
  <title>Kappa 분석 이해하기</title>
  <dc:creator>YeJi Kang</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-03-18-Kappa/</link>
  <description><![CDATA[ 
<!-- Google tag (gtag.js) -->
<script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script>





<section id="시작하기-전에" class="level1">
<h1>시작하기 전에</h1>
<p><a href="https://en.wikipedia.org/wiki/Cohen%27s_kappa">Kappa 통계</a>는 <strong>두 명 이상의 평가자(rater)가 범주형 데이터를 얼마나 일관되게 평가하는지를 측정하는 방법</strong>이다. 단순한 일치율과 달리, Kappa 계수는 무작위로 일치할 가능성을 보정하여 보다 신뢰할 수 있는 평가 일치도를 제공한다.이는 평가자의 주관적인 판단이 개입되는 연구에서 필수적이라고 볼 수 있다.</p>
<p>Kappa 분석에는 여러 가지 변형이 있으며, 상황에 따라 적절한 방법을 선택해야 한다. 대표적인 Kappa 분석 방법은 다음과 같다:</p>
<ol type="1">
<li><strong>Cohen’s Kappa (κ)</strong>: 두 명의 평가자가 범주형 데이터를 평가할 때 사용</li>
<li><strong>Cohen’s Weighted Kappa</strong>: 평가자 간의 불일치 정도를 가중치로 고려할 때 사용 (순위형 변수)</li>
<li><strong>Fleiss’ Kappa</strong>: 두 명 이상의 평가자가 있을 때 사용 (범주형 변수)</li>
<li><strong>Generalized Fleiss’ Kappa</strong>: Fleiss’ Kappa의 확장판으로, 순위형 데이터를 다룰 때 사용</li>
<li><strong>Krippendorff’s Alpha</strong>: 범주형, 순위형, 연속형 데이터 모두 적용 가능</li>
</ol>
<p>Cohen’s와 Cohen’s Weighted는 평가자가 두 명일때 적용되고, Fleiss’와 Generalized Fleiss’는 두 명 이상일 때 사용된다.</p>
<p>이 글에서는 R을 활용하여 다양한 Kappa 분석 방법을 구현하는 방법을 설명한다.</p>
<section id="단순-일치율-vs.-kappa-계수" class="level2">
<h2 class="anchored" data-anchor-id="단순-일치율-vs.-kappa-계수">단순 일치율 vs.&nbsp;Kappa 계수</h2>
<p>단순한 일치율(Percent Agreement)은 평가자 간의 동일한 판단이 나온 비율을 계산하는 방식이다. 하지만, 이는 무작위로 일치한 경우도 포함하기 때문에 신뢰도가 낮을 수 있다.</p>
<p>예를 들어, 두 평가자가 100개의 사례를 평가했을 때, 70개에서 동일한 판단을 내렸다면 단순 일치율은 70%이다. 하지만, 무작위로도 70%의 일치가 나올 가능성이 있다면, 실제 평가자의 일치 정도를 과대평가할 수 있다.</p>
<p>이를 보완하기 위해 Kappa 계수(<img src="https://latex.codecogs.com/png.latex?%CE%BA">)는 무작위 일치율(Expected Agreement)을 고려하여 조정된 값을 제공한다. 즉, Kappa 계수는 실제 일치율과 무작위 일치율 간의 차이를 기반으로 계산되며, 0~1 사이의 값으로 표현된다.</p>
<ul>
<li>Kappa 계수(<img src="https://latex.codecogs.com/png.latex?%CE%BA">) 해석:
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%CE%BA"> = 1: 완벽한 일치</li>
<li><img src="https://latex.codecogs.com/png.latex?%CE%BA"> = 0: 무작위 일치 수준</li>
<li><img src="https://latex.codecogs.com/png.latex?%CE%BA"> &lt; 0: 평가자가 오히려 무작위보다 더 불일치</li>
<li>0.6 ≤ <img src="https://latex.codecogs.com/png.latex?%CE%BA"> ≤ 0.8: 상당한 일치</li>
<li>0.4 ≤ <img src="https://latex.codecogs.com/png.latex?%CE%BA"> &lt; 0.6: 중간 수준의 일치</li>
</ul></li>
</ul>
<p>따라서 <img src="https://latex.codecogs.com/png.latex?%CE%BA">의 값은 크면 클수록 좋은 것이라고 본다.</p>
<p>이제 Kappa 분석이 왜 중요한지를 이해했으므로, 다음 섹션에서는 각 Kappa 분석 방법을 살펴보고, R을 이용하여 실제 데이터를 분석하는 방법을 설명한다.</p>
</section>
</section>
<section id="cohens-kappa" class="level1">
<h1>1. Cohen’s Kappa</h1>
<p>Cohen’s Kappa(<img src="https://latex.codecogs.com/png.latex?%CE%BA">)는 <strong>두 명의 평가자가 범주형 데이터를 평가할 때</strong> 사용되는 가장 기본적인 Kappa 계수이다. 여기서 측정되는 범주형 변수에서는 서로 다른 특성을 구분할 수 있지만, 이 특성 간의 순위나 서열 관계는 존재하지 않는다.</p>
<section id="cohens-kappa-공식" class="level2">
<h2 class="anchored" data-anchor-id="cohens-kappa-공식">1.1 Cohen’s Kappa 공식</h2>
<p>Cohen’s Kappa는 다음과 같은 공식으로 계산된다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%CE%BA%20=%20%5Cfrac%7BP_o%20-%20P_e%7D%7B1%20-%20P_e%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?P_o">: 평가자간 일치 확률 (Observed Accuracy)</li>
<li><img src="https://latex.codecogs.com/png.latex?P_e">: 우연히 일치된 평가를 받을 비율 (Expected Accuracy)</li>
</ul>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?P_e">는 ’우연히 일치할 확률’을 나타낸다. 예를 들어, 두 명의 상담사가 환자에게 우울증이 있는지 없는지에 대해 완전히 무작위로 판단했다고 가정한다. 마치 동전을 던지는 것처럼 말이다. 그럼에도 불구하고 두 상담사가 우연히 같은 결론을 내릴 가능성이 어느 정도 존재하게 되는데, 이 우연에 의한 일치 확률을 나타내는 값이 바로 <img src="https://latex.codecogs.com/png.latex?P_e">가 된다. 즉, <img src="https://latex.codecogs.com/png.latex?P_e">는 평가자들이 실제로 동의한 정도가 아니라, 순전히 우연으로 평가 결과가 같아질 가능성을 나타내는 가상의 확률이라고 이해하면 된다. <img src="https://latex.codecogs.com/png.latex?P_e">의 비율이 높을 수록 우연하게 일치한다는 것이고, 이 값이 최소에 가까워질수록 높은 <img src="https://latex.codecogs.com/png.latex?%CE%BA">의 값을 얻을 수 있게 된다.</p>
<section id="observed-accuracy-p_o" class="level3">
<h3 class="anchored" data-anchor-id="observed-accuracy-p_o">Observed Accuracy <img src="https://latex.codecogs.com/png.latex?P_o"></h3>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_0%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bg%7D%20f_%7Bii%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?n">: 전체 평가 개수</li>
<li><img src="https://latex.codecogs.com/png.latex?g">: 평가 범주의 개수 (예: 3개의 등급, 5개의 점수 등)</li>
<li><img src="https://latex.codecogs.com/png.latex?f_%7Bii%7D">: 평가자 두 명이 동일한 범주를 선택한 횟수</li>
</ul>
<p>즉, <img src="https://latex.codecogs.com/png.latex?P_0">는 전체 평가 중에서 평가자들이 동일한 범주를 선택한 비율을 의미한다. 이는 실제 데이터에서 평가자들이 얼마나 일치했는지를 보여주는 값이다.</p>
</section>
<section id="expected-accuracy-p_e" class="level3">
<h3 class="anchored" data-anchor-id="expected-accuracy-p_e">Expected Accuracy <img src="https://latex.codecogs.com/png.latex?P_e"></h3>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_e%20=%20%5Cfrac%7B1%7D%7Bn%5E2%7D%20%5Csum_%7Bi=1%7D%5E%7Bg%7D%20f_%7Bi+%7D%20f_%7B+i%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?f_%7Bi+%7D">: 특정 범주의 행 합 (첫 번째 평가자가 해당 범주를 선택한 횟수)</li>
<li><img src="https://latex.codecogs.com/png.latex?f_%7B+i%7D">: 특정 범주의 열 합 (두 번째 평가자가 해당 범주를 선택한 횟수)</li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?P_e">는 평가자들이 무작위로 평가했을 때 동일한 범주를 선택할 확률을 의미한다. 이는 두 평가자가 특정 범주를 선택할 확률을 각각 곱하여 계산되며, 모든 범주에 대해 합산하여 전체적인 기대 일치도를 구하는 방식이다.</p>
</section>
<section id="binary-classifications" class="level3">
<h3 class="anchored" data-anchor-id="binary-classifications">Binary Classifications</h3>
<p>Cohen’s Kappa는 이진 분류에서도 모델의 예측 신뢰도를 평가하는 중요한 지표로 활용될 수 있다. 이는 다음과 같은 공식으로 표현된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%CE%BA%20=%20%5Cfrac%7B2%20%5Ctimes%20(TP%20%5Ctimes%20TN%20-%20FN%20%5Ctimes%20FP)%7D%0A%7B(TP%20+%20FP)%20%5Ctimes%20(FP%20+%20TN)%20+%20(TP%20+%20FN)%20%5Ctimes%20(FN%20+%20TN)%7D%0A"></p>
<p>여기서 각 항목은 다음과 같은 의미를 가진다:</p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?TP"> (True Positives): 실제로 긍정(positive)인 경우를 정확하게 예측한 수</li>
<li><img src="https://latex.codecogs.com/png.latex?FP"> (False Positives): 실제로는 부정(negative)이지만, 긍정으로 잘못 예측한 수</li>
<li><img src="https://latex.codecogs.com/png.latex?TN"> (True Negatives): 실제로 부정인 경우를 정확하게 예측한 수</li>
<li><img src="https://latex.codecogs.com/png.latex?FN"> (False Negatives): 실제로는 긍정이지만, 부정으로 잘못 예측한 수</li>
</ul>
<p>이진 분류에서 Cohen’s Kappa는 단순한 정확도보다는 무작위 예측과 비교하여 모델이 얼마나 신뢰할 만한지를 측정하는 데 유용하다. 예를 들어, 불균형한 데이터에서 단순한 정확도는 높은 값이 나올 수 있지만, Kappa 값이 낮게 나오는 경우가 있을 수 있다. 이는 모델이 특정 클래스를 과도하게 예측하고 있음을 나타낼 수 있다.</p>
</section>
</section>
<section id="example-1" class="level2">
<h2 class="anchored" data-anchor-id="example-1">1.2 Example 1</h2>
<p>첫 번째 예시에서는 두 평가자가 다섯 개 항목을 각각 평가한 뒤, Cohen’s Kappa 통계량을 이용해 두 평가자 간의 일치도를 분석했다. R에서 Cohen’s Kappa를 구하기 위해서는 irr 패키지를 사용한다 (Desc Tools, psych 등 다른 패키지도 존재).</p>
<pre><code>library(irr)

# 두 평가자가 5개의 항목을 평가
ratings &lt;- data.frame(
  rater1 = c("A", "A", "B", "A", "C"),
  rater2 = c("A", "B", "B", "A", "C")
)

# Cohen's Kappa 계산
result &lt;- kappa2(ratings, weight = "unweighted")
print(result)</code></pre>
<p><strong>출력:</strong></p>
<pre><code> Cohen's Kappa for 2 Raters (Weights: unweighted)

 Subjects = 5 
   Raters = 2 
    Kappa = 0.688 

        z = 2.28 
  p-value = 0.0224</code></pre>
<p>이 예시의 결과는 Kappa 값이 0.688로 나타났고, z 통계량은 2.28, p값은 0.0224로 나타나 통계적으로 유의미한 일치도를 보였다. 이는 두 평가자가 단순히 우연히 일치하는 것을 넘어 실제로 상당히 일치된 평가를 내렸다는 의미로 해석할 수 있다.</p>
</section>
<section id="example-2" class="level2">
<h2 class="anchored" data-anchor-id="example-2">1.3 Example 2</h2>
<p>다음 예시는 실제 의료 환경에서 자주 발생하는 사례를 바탕으로 Cohen’s Kappa를 적용한 것이다. 두 명의 영상의학 전문의가 CT, MRI, PET, X-ray 총 4가지 영상진단 방식으로 환자를 각각 독립적으로 평가했을 때, 두 전문의 간 평가가 얼마나 일치하는지를 분석한다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(ggplot2); <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(officer)</span>
<span id="cb3-2"></span>
<span id="cb3-3">imaging.modalities <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"CT"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"MRI"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"PET"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Xray"</span>)</span>
<span id="cb3-4"></span>
<span id="cb3-5">kappa.results <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sapply</span>(imaging.modalities, <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(modality){</span>
<span id="cb3-6">  var1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(modality, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_Rater1"</span>)</span>
<span id="cb3-7">  var2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(modality, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"_Rater2"</span>)</span>
<span id="cb3-8"></span>
<span id="cb3-9">  kappa_calc <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> irr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kappa2</span>(diagnosis_data[, .SD, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">.SDcols =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(var1, var2)], <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">weight =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"unweighted"</span>)</span>
<span id="cb3-10">  standard_error <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> kappa_calc<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> kappa_calc<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>statistic</span>
<span id="cb3-11">  conf_interval <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(kappa_calc<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">qnorm</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.975</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> standard_error,</span>
<span id="cb3-12">                     kappa_calc<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">qnorm</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.975</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> standard_error)</span>
<span id="cb3-13"></span>
<span id="cb3-14">  <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">return</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(kappa_calc<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" (95% CI: "</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(conf_interval[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>], <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>),</span>
<span id="cb3-15">                <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"-"</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(conf_interval[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>], <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span>))</span>
<span id="cb3-16">})</span></code></pre></div></div>
<p>이 예시에서 계산된 Kappa 값은 각 영상진단 방식별로 제공되며, 각 값에 대한 표준오차와 95% 신뢰구간도 함께 계산된다. irr::kappa2 함수를 통해 두 평가자 간의 Cohen’s Kappa 값을 계산했고, weight = “unweighted” 옵션으로 평가 항목 간의 차이에 동일한 가중치를 부여한다.</p>
<p>이와 같이 Cohen’s Kappa는 두 평가자가 같은 값을 측정했는지 여부를 고려하지만, 불일치의 정도는 고려하지 않는다. Example 1을 보면 Cohen’s Kappa는 평가자가 A와 C를 선택했을 때와 A와 B를 선택했을 때를 동일한 불일치로 간주하고 있는 것을 확인할 수 있다. 그렇다면 범주형 변수가 아닌 순위형 변수가 있다면 어떨까?</p>
</section>
</section>
<section id="weighted-kappa" class="level1">
<h1>2. Weighted Kappa</h1>
<p>순위형 변수, 즉 특성을 정렬할 수 있는 변수가 있다면 그 순서 또한 고려해야 한다.</p>
<p>예를 들어, ‘불만족’, ’중립’과 ’만족’이 있다고 하자. 불만족과 중립 사이에는 불만족과 만족 사이보다 작은 차이가 있다. 이러한 차이를 고려하는 것이 바로 Weighted Kappa다.</p>
<p>다시 말해, Weighted Kappa는 <strong>순위형 데이터를 평가하는 두 명의 평가자 간 일치도를 측정하는 방법</strong>이다. Cohen’s Kappa은 일치 vs.&nbsp;불일치를 1과 0으로 구분하는 반면, Weighted Kappa는 평가자의 불일치 정도에 가중치를 부여하여 더 세밀한 분석이 가능하다.</p>
<section id="weighted-kappa-공식" class="level2">
<h2 class="anchored" data-anchor-id="weighted-kappa-공식">2.1 Weighted Kappa 공식</h2>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ckappa_w%20=%20%5Cfrac%7BP_o%20-%20P_e%7D%7B1%20-%20P_e%7D%0A"></p>
<p>이 식은 일반적인 Cohen’s Kappa의 공식과 같지만, Weighted Kappa의 경우 Observed Accuracy <img src="https://latex.codecogs.com/png.latex?P_o">와 Expected Accuracy <img src="https://latex.codecogs.com/png.latex?P_e">을 계산할 때 가중치를 적용한 값을 사용한다는 점에서 차이가 있다. 각 항목은 다음과 같이 계산한다.</p>
<section id="observed-accuracy-p_o-1" class="level3">
<h3 class="anchored" data-anchor-id="observed-accuracy-p_o-1">Observed Accuracy (<img src="https://latex.codecogs.com/png.latex?P_o">)</h3>
<p><img src="https://latex.codecogs.com/png.latex?P_o">는 두 평가자의 실제 평가 결과를 바탕으로 각 범주 간에 가중치를 적용하여 계산한 값이다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_%7Bo%7D%20=%20%5Csum_%7Bi%7D%5Csum_%7Bj%7D%20W_%7Bij%7DP_%7Bij%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?W_%7Bij%7D"> : 각 범주(i,j) 간의 가중치</li>
<li><img src="https://latex.codecogs.com/png.latex?P_%7Bij%7D"> : 두 평가자가 범주 (i,j)를 선택한 관측 비율</li>
</ul>
</section>
<section id="expected-accuracy-p_e-1" class="level3">
<h3 class="anchored" data-anchor-id="expected-accuracy-p_e-1">Expected Accuracy (<img src="https://latex.codecogs.com/png.latex?P_e">)</h3>
<p><img src="https://latex.codecogs.com/png.latex?P_e">는 각 평가자의 범주별 평가 확률의 곱에 가중치를 곱하여 계산된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_e%20=%20%5Csum_%7Bi%7D%5Csum_%7Bj%7D%20W_%7Bij%7DP_%7Bi+%7DP_%7B+j%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?P_%7Bi+%7D"> : 평가자 1이 범주 i를 선택한 전체 비율 (행 방향)</li>
<li><img src="https://latex.codecogs.com/png.latex?P_%7B+j%7D"> : 평가자 2가 범주 j를 선택한 전체 비율 (열 방향)</li>
</ul>
</section>
</section>
<section id="가중치w의-종류" class="level2">
<h2 class="anchored" data-anchor-id="가중치w의-종류">2.2 가중치(<img src="https://latex.codecogs.com/png.latex?W">)의 종류</h2>
<p>Weighted Kappa에서 주로 사용하는 대표적인 가중치 부여 방식은 두 가지다: 선형(linear)과 제곱(quadratic)</p>
<section id="선형-가중치-linear-weights" class="level3">
<h3 class="anchored" data-anchor-id="선형-가중치-linear-weights">선형 가중치 (Linear weights)</h3>
<p>선형 가중치는 <strong>Cicchetti-Allison weights</strong>라고도 하며, 평가 항목 간의 불일치 정도에 따라 <strong>일정한 간격으로</strong> 가중치를 부여한다. 즉, 두 평가자 간의 평가가 한 단계씩 멀어질 때마다 일정한 비율로 일치도가 감소한다. 특징은 각 평가 간의 차이에 비례하여(선형적으로) 가중치를 부여한다는 것이다.</p>
<p><strong>수식 표현:</strong></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AW_%7Bij%7D%5E%7Blinear%7D%20=%201%20-%20%5Cfrac%7B%7Ci%20-%20j%7C%7D%7Bk%20-%201%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?i">, <img src="https://latex.codecogs.com/png.latex?j">: 두 평가자가 선택한 범주(단계)</li>
<li><img src="https://latex.codecogs.com/png.latex?k">: 범주의 전체 개수</li>
</ul>
<p><strong>예시:</strong><br>
범주가 1~4단계로 구성된 경우 (<img src="https://latex.codecogs.com/png.latex?k">=4):</p>
<ul>
<li>평가자가 각각 1단계와 2단계를 선택했다면:</li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?%0AW_%7B12%7D%5E%7Blinear%7D%20=%201%20-%20%5Cfrac%7B%7C1%20-%202%7C%7D%7B4%20-%201%7D%20=%20%5Cfrac%7B2%7D%7B3%7D%20%5Capprox%200.67%0A"></p>
<p>이는 평가자들이 약 67% 정도로 일치하고 있다고 볼 수 있으며, 33%는 불일치한다고 해석할 수 있다. 이 결과를 제곱 가중치를 적용했을 때와 비교해 본다.</p>
</section>
<section id="제곱-가중치-quadratic-weights" class="level3">
<h3 class="anchored" data-anchor-id="제곱-가중치-quadratic-weights">제곱 가중치 (Quadratic weights)</h3>
<p>제곱 가중치는 <strong>Fleiss-Cohen weights</strong>라고도 하며, 평가 항목 간의 불일치가 클수록 가중치를 <strong>제곱에 비례하여(비선형적으로)</strong> 부여한다. 특히 평가자 간의 작은 불일치는 가볍게, 큰 불일치는 더 무겁게 여긴다. 즉, 불일치의 정도가 증가할수록 가중치는 비선형적으로(제곱의 비율로) 감소하게 된다.</p>
<p><strong>수식 표현:</strong></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AW_%7Bij%7D%5E%7Bquadratic%7D%20=%201%20-%20%5Cfrac%7B(i%20-%20j)%5E2%7D%7B(k%20-%201)%5E2%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?i">, <img src="https://latex.codecogs.com/png.latex?j">: 두 평가자가 선택한 범주(단계) - (<img src="https://latex.codecogs.com/png.latex?k">): 범주의 전체 개수</li>
</ul>
<p><strong>예시:</strong><br>
Linear의 예시와 마찬가지로 범주가 1~4단계로 구성된 경우(<img src="https://latex.codecogs.com/png.latex?k">=4):</p>
<ul>
<li>평가자가 각각 1단계와 2단계를 선택했다면:</li>
</ul>
<p><img src="https://latex.codecogs.com/png.latex?%0AW_%7B12%7D%5E%7Bquadratic%7D%20=%201%20-%20%5Cfrac%7B(1%20-%202)%5E2%7D%7B(4%20-%201)%5E2%7D%20=%201%20-%20%5Cfrac%7B1%7D%7B9%7D%20=%20%5Cfrac%7B8%7D%7B9%7D%20%5Capprox%200.89%0A"></p>
<p>이는 평가자들이 약 89%로 상당히 높은 일치도를 나타내며, 한 단계만 차이가 나기에 작은 불일치 정도에 높은 가중치를 부여한 것이다. 그러나 두 단계 이상의 큰 차이가 발생하면 가중치가 급격히 감소하여, 큰 불일치로 인식을 한다.</p>
<p>이와 같이 Quadratic 방식에서는 큰 불일치를 더욱 엄격하게 평가한다.</p>
</section>
<section id="가중치-선택-방법" class="level3">
<h3 class="anchored" data-anchor-id="가중치-선택-방법">가중치 선택 방법</h3>
<ul>
<li><strong>선형 가중치(Linear weights)</strong>는 모든 단계 간의 차이를 동일한 중요도로 평가할 때 사용한다.
<ul>
<li>진단 결과가 1단계에서 2단계로 바뀌는 것과, 2단계에서 3단계로 바뀌는 것의 중요도가 같은 경우</li>
</ul></li>
<li><strong>제곱 가중치(Quadratic weights)</strong>는 작은 차이보다 큰 차이에 더 큰 중요성을 부여할 때 적합하다.
<ul>
<li>1단계와 2단계 간의 차이는 그리 크지 않지만, 2단계와 3단계 간의 차이는 매우 큰 의미를 가지는 경우</li>
</ul></li>
</ul>
<p>결국, 분석하려는 데이터와 평가 기준의 특성에 따라 적절한 가중치를 선택하면 된다.</p>
</section>
</section>
<section id="example-1-1" class="level2">
<h2 class="anchored" data-anchor-id="example-1-1">2.3 Example 1</h2>
<p>Weighted Kappa는 R코드를 활용해 수월하게 구할 수 있다.</p>
<p>아래는 두 명의 영상의학 전문의가 MRI 영상을 보고 병변의 심각도를 5단계 척도(1: 정상, 2: 경미, 3: 중등도, 4: 심함, 5: 매우 심함)로 평가한 경우를 예로 든 것이다. 두 평가자 간의 일치도를 Weighted Kappa를 사용하여 분석하며, 선형(linear) 가중치를 적용하여 평가 간의 차이를 부분적으로 반영한다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(irr)</span>
<span id="cb4-2"></span>
<span id="cb4-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 평가 데이터 예시 (랜덤하게 생성)</span></span>
<span id="cb4-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">123</span>)</span>
<span id="cb4-5">lesion_assessment <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb4-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Rater1 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">replace =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>),</span>
<span id="cb4-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">Rater2 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">sample</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">replace =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>)</span>
<span id="cb4-8">)</span>
<span id="cb4-9"></span>
<span id="cb4-10"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 두 평가자 간의 Weighted Cohen's Kappa</span></span>
<span id="cb4-11">kap <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> irr<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kappa2</span>(lesion_assessment, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">weight =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"equal"</span>)</span>
<span id="cb4-12">se <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> kap<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> kap<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>statistic</span>
<span id="cb4-13">ci_lower <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> kap<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">qnorm</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.975</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> se</span>
<span id="cb4-14">ci_upper <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> kap<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">qnorm</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.975</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> se</span>
<span id="cb4-15"></span>
<span id="cb4-16">weighted_kappa_result <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">paste0</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(kap<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>value, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">" (95% CI: "</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(ci_lower, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"~"</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">round</span>(ci_upper, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>), <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">")"</span>)</span>
<span id="cb4-17">weighted_kappa_result</span></code></pre></div></div>
<p>이 예시를 통해 두 평가자 간의 의견 일치 수준을 파악할 수 있다.위 코드 실행 결과로 선형 가중치를 사용한 Weighted Kappa 값을 확인할 수 있으며, 범주 간의 불일치 정도를 반영한 평가자 간 일치도를 평가할 수 있다. 여기서 quadratic 가중치를 부여하기 위해서는 weight = “squared”로 수정하면 된다.</p>
<p><strong>출력:</strong></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">[<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>] <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"0.048 (95% CI: -0.089~0.185)"</span></span></code></pre></div></div>
<p>처음에 설명했던 것 처럼, Cohen’s Kappa 값이 0.048이라는 것은 두 평가자가 거의 무작위로 평가를 했거나, 일치도가 매우 낮다는 것을 의미한다. Example의 결과 값으로 0.048로 나타났기 때문에 이런 경우 평가자 간의 의견이 거의 일치하지 않는다고 결론을 내린다.</p>
<p>신뢰구간을 살펴보면 -0.089에서 0.185 사이에 위치하고 있다. 95% 신뢰구간은 실제 모집단에서의 Kappa 값이 이 범위 안에 있을 확률이 95%라는 뜻이다. 하지만 이 신뢰구간에는 음수 값(-0.089)이 포함되어 있으므로 일반적으로 평가자 간의 일치도가 단순한 우연보다도 낮다는 뜻이다. 이는 평가자들이 무작위로 답변한 것보다도 더 일치도가 낮을 가능성이 있다는 것이며, 신뢰구간이 넓다는 것은 데이터가 불안정하거나, 평가자 간의 일치도가 일정하지 않다는 것을 나타낸다.</p>
<p>예시에서 이러한 결과가 나오는 이유는 평가 데이터 자체가 랜덤하게 생성되었기 때문이다. 그러므로 당연히 통계적으로 신뢰하기 어려운 결과가 나온다. 실제 상황에서 이러한 Kappa 값이 나온다면, 평가 기준을 명확하게 정리하고, 데이터의 질을 개선하는 것이 필요할 것이다.</p>
</section>
</section>
<section id="fleiss-kappa" class="level1">
<h1>3. Fleiss’ Kappa</h1>
<p>앞서 살펴본 Cohen’s와 Weighted Kappa는 평가자 또는 연구자가 정확히 두 명일 때만 적용이 가능하다. 그렇다면 평가자 수가 세 명 이상일 경우에는 어떻게 해야 하는가. 이런 상황을 해결하기 위해 다양한 접근이 제안되었다. 평가자들을 두 명씩 묶어 모든 조합의 Cohen’s kappa 값을 구한 뒤 그 평균을 사용하는 방법이 존재하기도 하나, 현재 학계에서 가장 폭넓게 사용되고 인정받는 방식은 Fleiss Kappa이다.</p>
<p><strong>Fleiss’ Kappa</strong>(<img src="https://latex.codecogs.com/png.latex?%5Ckappa">)는 세 명 이상의 평가자가 범주형 데이터를 평가할 때, 평가자 간의 전반적인 <strong>일치도(agreement)</strong> 를 측정하는 통계 지표다. Cohen’s Kappa가 두 명의 평가자에 대해서만 사용되는 반면, Fleiss’ Kappa는 다수의 평가자가 있는 경우에도 적용이 가능하다.</p>
<p>또한, Fleiss’ Kappa는 한 명의 평가자가 같은 항목을 <strong>두 번 이상 서로 다른 시점에서 반복하여 평가</strong>할 때도 사용할 수 있다. 이 경우 Fleiss’ Kappa는 동일한 평가자가 여러 번 평가했을 때, 각각의 평가가 얼마나 일관성 있게 일치하는지를 나타내는 지표가 된다. 즉, 동일 평가자의 시간에 따른 평가 일관성을 측정하는 데에도 Fleiss’ Kappa가 활용될 수 있는 것이다.</p>
<section id="fleiss-kappa-공식" class="level2">
<h2 class="anchored" data-anchor-id="fleiss-kappa-공식">3.1 Fleiss’ Kappa 공식</h2>
<p>Fleiss’ Kappa(<img src="https://latex.codecogs.com/png.latex?%5Ckappa">)를 정의하는 수식은 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ckappa%20=%20%5Cfrac%7B%5Cbar%7BP_o%7D%20-%20%5Cbar%7BP_e%7D%7D%7B1%20-%20%5Cbar%7BP_e%7D%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP_o%7D">: 관찰된 평균 일치율 (Observed agreement)</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP_e%7D">: 우연에 의한 평균 일치율 (Expected agreement)</li>
</ul>
<p>이와 같이 Fleiss’ Kappa는 관찰된 평균 일치율(<img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP%7D_0">)과 우연에 의한 평균 일치율(<img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP%7D_e">)을 사용하여 계산한다. 각각의 공식은 아래와 같다.</p>
<section id="observed-agreement-barp_o" class="level3">
<h3 class="anchored" data-anchor-id="observed-agreement-barp_o">Observed agreement <img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP%7D_o">:</h3>
<p><img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP%7D_o">는 평가자들이 실제로 얼마나 의견이 일치했는지를 측정하는 값인데, 이는 실제 평가에서 동일한 범주를 선택한 평가자들의 비율을 정량적으로 나타내는 값이며, 다음과 같은 수식으로 계산된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbar%7BP%7D_o%20=%20%5Cfrac%7B1%7D%7BN%20n%20(n%20-%201)%7D%5Cleft(%5Csum_%7Bi=1%7D%5E%7BN%7D%5Csum_%7Bj=1%7D%5E%7Bk%7Dn_%7Bij%7D%5E%7B2%7D%20-%20N%20n%5Cright)%0A"> - <img src="https://latex.codecogs.com/png.latex?N">: 평가 항목의 총 개수 - <img src="https://latex.codecogs.com/png.latex?n">: 각 대상당 평가자의 수 - <img src="https://latex.codecogs.com/png.latex?k">: 평가 범주의 수 - <img src="https://latex.codecogs.com/png.latex?n_%7Bij%7D">: <img src="https://latex.codecogs.com/png.latex?i">번째 대상에서 <img src="https://latex.codecogs.com/png.latex?j">번째 카테고리를 선택한 평가자의 수</p>
<p>이 공식에서 첫 번째 항인 <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%5Csum_%7Bj=1%7D%5E%7Bk%7Dn_%7Bij%7D%5E%7B2%7D">는 특정 범주를 선택한 평가자 수를 제곱하고 합산한다. 동일한 범주를 선택한 평가자가 많을수록 그 값이 더 커지며, 평가자들이 일관된 결정을 내릴수록 이 항의 값이 증가하게 된다. 이 식에서 <img src="https://latex.codecogs.com/png.latex?Nn">을 빼는 이유는, <img src="https://latex.codecogs.com/png.latex?%5Csum%20n_%7Bij%7D%5E2"> 항에는 평가자가 한 명만 특정 범주를 선택한 경우도 포함되기 때문이다. 평가자 수가 많을수록 이 값이 증가하므로, 이를 보정하기 위해 전체 평가 대상 개수(<img src="https://latex.codecogs.com/png.latex?N">)와 평가자 수(<img src="https://latex.codecogs.com/png.latex?n">)를 곱한 값을 빼준다. 이를 통해, 평가자가 많을수록 발생할 수 있는 단순한 일치 효과를 제거하고, 실제로 의미 있는 평가자 간의 일치도를 측정할 수 있도록 조정한다.</p>
<p>마지막으로, 이 값을 <img src="https://latex.codecogs.com/png.latex?N%20n%20(n%20-%201)">로 나누어 정규화한다. 이는 전체 평가 수에 대해 평균을 구하는 과정이며, 결과적으로 Observed Agreement는 “평균적인 일치도”를 나타내는 값이 된다. 이 값이 클수록 평가자들이 동일한 평가를 내린 비율이 높다는 것을 의미하며, 평가자 간의 의견이 더욱 일관되게 나타난다는 것을 보여준다. 이 값은 Expected Agreement(<img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP%7D_e">)와 비교하여 Fleiss’ Kappa 값을 계산하는 핵심 요소다.</p>
</section>
<section id="expected-agreement-by-chance-barp_e" class="level3">
<h3 class="anchored" data-anchor-id="expected-agreement-by-chance-barp_e">Expected agreement by chance <img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP_e%7D"></h3>
<p><img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP_e%7D">은 평가자들이 평가를 무작위로 진행했을 때 결과가 우연히 일치하게 될 확률을 나타낸다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbar%7BP%7D_e%20=%20%5Csum_%7Bj=1%7D%5E%7Bk%7D%20P_j%5E%7B2%7D%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?P_j">는 모든 대상과 평가자를 통틀어 <img src="https://latex.codecogs.com/png.latex?j">번째 카테고리에 선택된 횟수의 합을 전체 평가 횟수(<img src="https://latex.codecogs.com/png.latex?N%20%5Ctimes%20n">)로 나눈 값이다. 이를 식으로 표현할 수 있다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_j%20=%20%5Cfrac%7B1%7D%7BNn%7D%20%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_%7Bij%7D%0A"> 이 공식에서 <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_%7Bij%7D">는 모든 평가 대상에서 특정 범주 <img src="https://latex.codecogs.com/png.latex?j">가 선택된 총 횟수를 의미하며, 이를 전체 평가 횟수(<img src="https://latex.codecogs.com/png.latex?N%20%5Ctimes%20n">)로 나누어 특정 범주 <img src="https://latex.codecogs.com/png.latex?j">가 선택될 확률을 구한다.</p>
<p>무작위로 평가를 했다고 가정하면, 한 평가자가 특정 범주 <img src="https://latex.codecogs.com/png.latex?j">를 선택할 확률은 <img src="https://latex.codecogs.com/png.latex?P_j">이고, 또 다른 평가자도 동일한 범주를 선택할 확률 역시 <img src="https://latex.codecogs.com/png.latex?P_j">이므로, 이 두 확률을 곱한 <img src="https://latex.codecogs.com/png.latex?P_j%5E2">이 해당 범주에서 평가가 우연히 일치할 확률이 된다. 따라서 모든 범주 <img src="https://latex.codecogs.com/png.latex?j=1">부터 <img src="https://latex.codecogs.com/png.latex?k">까지에 대해 이러한 우연 일치 확률을 더하면, 전체적으로 평가자들이 무작위로 평가했을 때 기대되는 평균적인 일치 확률 <img src="https://latex.codecogs.com/png.latex?%5Cbar%7BP%7D_e">을 계산할 수 있다.</p>
</section>
</section>
<section id="example" class="level2">
<h2 class="anchored" data-anchor-id="example">3.2 Example</h2>
<p>다음 예시 데이터로 Fleiss’ Kappa의 개념을 살펴본다.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>항목</th>
<th>범주1</th>
<th>범주2</th>
<th>범주3</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>1</td>
<td>0</td>
<td>0</td>
<td>5</td>
</tr>
<tr class="even">
<td>2</td>
<td>0</td>
<td>1</td>
<td>4</td>
</tr>
<tr class="odd">
<td>3</td>
<td>1</td>
<td>0</td>
<td>4</td>
</tr>
<tr class="even">
<td>4</td>
<td>0</td>
<td>2</td>
<td>3</td>
</tr>
<tr class="odd">
<td>5</td>
<td>0</td>
<td>1</td>
<td>4</td>
</tr>
</tbody>
</table>
<ul>
<li>평가 항목 수: <img src="https://latex.codecogs.com/png.latex?N%20=%205"></li>
<li>평가자 수: <img src="https://latex.codecogs.com/png.latex?n%20=%205"> (각 항목마다 평가자가 5명)</li>
<li>평가 범주 수: <img src="https://latex.codecogs.com/png.latex?k%20=%203"></li>
</ul>
<p><strong>R 코드로 구하는 방식:</strong></p>
<pre><code># `irr` 패키지의 `kappam.fleiss()` 함수를 사용한다
install.packages("irr")
library(irr)

# 데이터 행렬 생성
ratings &lt;- matrix(c(
  0, 0, 5,
  0, 1, 4,
  1, 0, 4,
  0, 2, 3,
  0, 1, 4), 
  nrow = 5, byrow = TRUE)

# Fleiss' Kappa 계산
fleiss_kappa &lt;- kappam.fleiss(ratings)
print(fleiss_kappa)</code></pre>
<p><strong>출력:</strong></p>
<pre><code> Fleiss' Kappa for m Raters

 Subjects = 5 
   Raters = 3 
    Kappa = -0.25 

        z = -1.83 
  p-value = 0.067 </code></pre>
<p>Cohen’s Kappa와 같이 Fleiss’ Kappa는 평가 값이 동일한 경우 1, 다르면 0으로 단순 비교한다. 순위형 변수로 Weight를 부여해야 할 경우, Generalized Fleiss’ Kappa를 사용하면 된다.</p>
</section>
</section>
<section id="generalized-fleiss-kappa" class="level1">
<h1>4. Generalized Fleiss’ Kappa</h1>
<p><strong>Generalized Fleiss’ Kappa</strong>는 Fleiss’ Kappa를 일반화한 지표로, 여러 명의 평가자가 평가하는 경우 사용된다. 각 평가자가 평가하는 항목 수가 다르거나 평가 범주 간 순서가 중요한 경우에도 사용할 수 있는 지표다. 이는 평가 값 사이의 거리를 반영하여 평가자 간의 불일치 정도를 측정한다.</p>
<p>기존 Fleiss’ Kappa와 달리 평가자가 모든 항목을 평가하지 않아도 되며, 항목마다 평가자 수가 달라도 적용할 수 있다. 또한 항목 간, 평가자 간의 부분적인 데이터 결측이 있더라도 신뢰도 높은 일치도를 구할 수 있다.</p>
<section id="generalized-fleiss-kappa-공식" class="level2">
<h2 class="anchored" data-anchor-id="generalized-fleiss-kappa-공식">4.1 Generalized Fleiss’ Kappa 공식</h2>
<p>Generalized Fleiss’ Kappa (<img src="https://latex.codecogs.com/png.latex?%5Ckappa_G">)의 수식은 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ckappa_%7BG%7D%20=%20%5Cfrac%7BP_o%20-%20P_e%7D%7B1%20-%20P_e%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?P_o">: 관찰된 가중 평균 일치율 (Observed weighted agreement)</li>
<li><img src="https://latex.codecogs.com/png.latex?P_e">: 우연히 기대되는 가중 평균 일치율 (Expected weighted agreement by chance)</li>
</ul>
<p>각 항목의 계산법은 아래와 같다.</p>
<section id="observed-weighted-agreement-p_o" class="level3">
<h3 class="anchored" data-anchor-id="observed-weighted-agreement-p_o">Observed weighted agreement <img src="https://latex.codecogs.com/png.latex?P_o"></h3>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_o%20=%20%5Cfrac%7B1%7D%7B%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i(n_i%20-%201)%7D%20%5Csum_%7Bi=1%7D%5E%7BN%7D%5Csum_%7Bj=1%7D%5E%7Bk%7D%5Csum_%7Bl=1%7D%5E%7Bk%7D%20W_%7Bjl%7D%20%5Ccdot%20n_%7Bij%7D(n_%7Bil%7D-%5Cdelta_%7Bjl%7D)%0A"> - <img src="https://latex.codecogs.com/png.latex?N">: 평가된 항목(대상)의 총 개수 - <img src="https://latex.codecogs.com/png.latex?k">: 평가 범주의 수 - <img src="https://latex.codecogs.com/png.latex?n_i">: <img src="https://latex.codecogs.com/png.latex?i">번째 항목을 평가한 평가자의 수 - <img src="https://latex.codecogs.com/png.latex?n_%7Bij%7D">: <img src="https://latex.codecogs.com/png.latex?i">번째 항목을 <img src="https://latex.codecogs.com/png.latex?j">번째 범주로 평가한 평가자의 수 - <img src="https://latex.codecogs.com/png.latex?W_%7Bjl%7D">: 범주 간 가중치 - <img src="https://latex.codecogs.com/png.latex?%5Cdelta_%7Bjl%7D">: 범주가 같으면 1, 다르면 0인 값</p>
<p>Generalized Fleiss’ Kappa에서 Observed Weighted Agreement <img src="https://latex.codecogs.com/png.latex?P_o">는 평가자들이 실제로 얼마나 일치했는지를 측정하는 값이다. 기존 Fleiss’ Kappa가 단순한 평가 일치율을 계산하는 방식이라면, Generalized Fleiss’ Kappa는 평가 값 사이의 차이를 반영하여 가중치를 적용하는 방식으로 평가자 간의 일치도를 보다 정교하게 측정한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?P_o">는 평가자들이 동일한 평가를 내린 정도를 가중(weighted) 방식으로 계산하며, 전체 평가 대상에서 발생한 모든 평가 쌍에 대한 가중 평균을 구하는 방식으로 정의된다. 먼저, 전체 평가자들이 내린 평가 쌍의 총 개수는 <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i(n_i%20-%201)">로 계산된다. 여기서 <img src="https://latex.codecogs.com/png.latex?N">은 평가 대상의 개수이며, <img src="https://latex.codecogs.com/png.latex?n_i">는 특정 평가 대상에 대해 평가를 수행한 평가자 수이다. 평가 대상마다 평가자 수가 다를 수 있으므로 이를 반영하여 전체적인 합을 계산한다.</p>
<p>식에서 그 외의 부분은 평가자 간의 일치도를 계산한다. <img src="https://latex.codecogs.com/png.latex?W_%7Bjl%7D">은 범주 <img src="https://latex.codecogs.com/png.latex?j">와 범주 <img src="https://latex.codecogs.com/png.latex?l"> 사이의 가중치로, 두 평가 값이 얼마나 다른지를 수치적으로 반영하는 역할을 하는데, 앞서 설명한 linear 또는 quadratic 방식으로 설정된다. <img src="https://latex.codecogs.com/png.latex?n_%7Bij%7D">는 평가 대상 <img src="https://latex.codecogs.com/png.latex?i">에서 범주 <img src="https://latex.codecogs.com/png.latex?j">를 선택한 평가자의 수를 의미하며, <img src="https://latex.codecogs.com/png.latex?n_%7Bil%7D">은 동일한 평가 대상에서 범주 <img src="https://latex.codecogs.com/png.latex?l">을 선택한 평가자의 수를 나타낸다. 또한 <img src="https://latex.codecogs.com/png.latex?%5Cdelta_%7Bjl%7D">은 Kronecker Delta로, <img src="https://latex.codecogs.com/png.latex?j">와 <img src="https://latex.codecogs.com/png.latex?l">이 동일한 경우 1, 다르면 0을 반환하는 지표이다.</p>
<p><img src="https://latex.codecogs.com/png.latex?P_o%20=%201">이면 평가자들이 완벽하게 동일한 평가를 내린 경우이며, <img src="https://latex.codecogs.com/png.latex?P_o%20=%200">이면 평가자 간 평가가 완전히 무작위로 이루어진 경우를 의미한다.</p>
</section>
<section id="expected-weighted-agreement-by-chance-p_e" class="level3">
<h3 class="anchored" data-anchor-id="expected-weighted-agreement-by-chance-p_e">Expected weighted agreement by chance <img src="https://latex.codecogs.com/png.latex?P_e"></h3>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_e%20=%20%5Cfrac%7B1%7D%7B%5Cleft(%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i%5Cright)%5E2%20-%20%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i%7D%20%5Csum_%7Bj=1%7D%5E%7Bk%7D%5Csum_%7Bl=1%7D%5E%7Bk%7D%20W_%7Bjl%7D%5Cleft(%5Csum_%7Bi=1%7D%5E%7BN%7Dn_%7Bij%7D%5Cright)%5Cleft(%5Csum_%7Bi=1%7D%5E%7BN%7Dn_%7Bil%7D-%5Cdelta_%7Bjl%7D%5Cright)%0A"> <img src="https://latex.codecogs.com/png.latex?P_e">**는 Fleiss’ Kappa에서 Expected agreement by chance (<img src="https://latex.codecogs.com/png.latex?P_e">)를 확장한 개념으로, 평가 값들 사이의 거리를 고려하여 가중(weighted) 방식으로 측정한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cleft(%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i%5Cright)%5E2%20-%20%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i">는 전체 평가 데이터에서 발생할 수 있는 모든 평가 쌍의 개수를 나타낸다. 여기서 <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i">는 전체 평가자가 수행한 총 평가 개수이며, 이를 제곱한 값에서 자기 자신과의 비교를 제외하기 위해 <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_i">를 빼준다. 이는 평가자들이 임의로 범주를 선택했을 때 가능한 모든 평가 쌍의 개수를 정규화하는 역할을 한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_%7Bij%7D">와 <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5E%7BN%7D%20n_%7Bil%7D">은 특정 범주 <img src="https://latex.codecogs.com/png.latex?j">와 <img src="https://latex.codecogs.com/png.latex?l">이 전체 평가에서 각각 몇 번 선택되었는지를 나타낸다. 이 값에 가중치 행렬 <img src="https://latex.codecogs.com/png.latex?W_%7Bjl%7D">을 곱하는데, <img src="https://latex.codecogs.com/png.latex?W_%7Bjl%7D">은 범주 <img src="https://latex.codecogs.com/png.latex?j">와 범주 <img src="https://latex.codecogs.com/png.latex?l"> 사이의 거리를 반영하는 값으로, linear 또는 quadratic 가중치로 결정된다.마지막으로, <img src="https://latex.codecogs.com/png.latex?%5Cdelta_%7Bjl%7D">는 두 범주가 동일할 경우 1, 다를 경우 0을 갖는 함수다.</p>
<p>결과적으로 <img src="https://latex.codecogs.com/png.latex?P_e">를 <img src="https://latex.codecogs.com/png.latex?P_o">와 비교하여 Kappa 값이 계산되며, <img src="https://latex.codecogs.com/png.latex?P_o">가 <img src="https://latex.codecogs.com/png.latex?P_e">보다 클수록 평가자 간의 신뢰도가 높다는 것을 의미한다.</p>
</section>
</section>
<section id="example-3" class="level2">
<h2 class="anchored" data-anchor-id="example-3">4.2 Example</h2>
<p>다음과 같은 예시 데이터를 통해 계산 방식을 간단히 이해해 본다. 평가 범주는 1~3으로 순서형이며, 일부 평가자는 특정 항목을 평가하지 않았다.</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>항목</th>
<th>평가자1</th>
<th>평가자2</th>
<th>평가자3</th>
<th>평가자4</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>1</td>
<td>1</td>
<td>2</td>
<td>2</td>
<td>-</td>
</tr>
<tr class="even">
<td>2</td>
<td>2</td>
<td>2</td>
<td>3</td>
<td>2</td>
</tr>
<tr class="odd">
<td>3</td>
<td>3</td>
<td>3</td>
<td>-</td>
<td>-</td>
</tr>
<tr class="even">
<td>4</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>2</td>
</tr>
</tbody>
</table>
<p>Generalized Fleiss’ Kappa는 R의 <code>irrCAC</code> 패키지에 있는 <code>fleiss.kappa.raw</code> 함수를 사용하여 쉽게 계산할 수 있다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(irrCAC)</span>
<span id="cb8-2"></span>
<span id="cb8-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 여러 평가자의 병변 평가 데이터</span></span>
<span id="cb8-4">ratings <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb8-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater1 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>),</span>
<span id="cb8-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater2 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>),</span>
<span id="cb8-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater3 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>),</span>
<span id="cb8-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater4 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb8-9">)</span>
<span id="cb8-10"></span>
<span id="cb8-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Linear Weighted Fleiss Generalized Kappa</span></span>
<span id="cb8-12">kappa_linear <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fleiss.kappa.raw</span>(ratings, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">weights =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"linear"</span>)</span>
<span id="cb8-13"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">print</span>(kappa_linear)</span>
<span id="cb8-14"></span>
<span id="cb8-15"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Quadratic Weighted Fleiss Generalized Kappa</span></span>
<span id="cb8-16">kappa_quadratic <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">fleiss.kappa.raw</span>(ratings, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">weights =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"quadratic"</span>)</span>
<span id="cb8-17"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">print</span>(kappa_quadratic)</span></code></pre></div></div>
</section>
</section>
<section id="krippendorffs-alpha" class="level1">
<h1>5. Krippendorff’s Alpha</h1>
<p>Krippendorff’s Alpha는 여러 평가자가 동일한 항목을 평가할 때, 평가자 간의 신뢰도를 측정하는 지표이다. 가장 큰 특징은 범주형, 순서형, 연속형 데이터를 모두 처리할 수 있으며, 평가자 수에 제한이 없고, 불완전한 데이터(NA 값 포함)도 분석할 수 있다는 점이다. 이러한 특성 덕분에 결측값이 포함된 데이터에서도 안정적인 신뢰도 평가가 가능하고, Cohen’s Kappa나 Fleiss Kappa보다 범용성이 뛰어나다.</p>
<p>범주형, 순서형, 연속형 데이터: - <strong>Nominal (명목형)</strong>: 범주 간 서열이 없는 경우 - <strong>Ordinal (순위형)</strong>: 서열이 있는 경우 (예: 경미, 중등, 심함) - <strong>Interval (구간형)</strong>: 연속형 데이터 (예: 온도, IQ 점수 등)</p>
<section id="신뢰도-데이터" class="level2">
<h2 class="anchored" data-anchor-id="신뢰도-데이터">5.1 신뢰도 데이터</h2>
<p>먼저 Krippendorff’s Alpha를 계산하기 위해서는 평가자들이 동일한 단위(unit)에 대해 내린 평가 데이터를 분석해야 한다. 평가자들은 독립적으로 하나 이상의 값을 할당할 수 있으며, 이러한 데이터를 <strong>m × N 행렬</strong>로 표현할 수 있다. 여기서 m은 평가자의 수, N은 평가된 항목의 개수이다.</p>
<p>평가자가 특정 단위 <img src="https://latex.codecogs.com/png.latex?u_j">에 할당한 값 <img src="https://latex.codecogs.com/png.latex?v_%7Bij%7D">를 포함하는 행렬을 다음과 같이 정의할 수 있다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Barray%7D%7Bc%7Ccccc%7D%0A%20%20%20%20&amp;%20u_1%20&amp;%20u_2%20&amp;%20u_3%20&amp;%20%5Ccdots%20&amp;%20u_N%20%5C%5C%5Chline%0Ac_1%20&amp;%20v_%7B11%7D%20&amp;%20v_%7B12%7D%20&amp;%20v_%7B13%7D%20&amp;%20%5Ccdots%20&amp;%20v_%7B1N%7D%20%5C%5C%0Ac_2%20&amp;%20v_%7B21%7D%20&amp;%20v_%7B22%7D%20&amp;%20v_%7B23%7D%20&amp;%20%5Ccdots%20&amp;%20v_%7B2N%7D%20%5C%5C%0Ac_3%20&amp;%20v_%7B31%7D%20&amp;%20v_%7B32%7D%20&amp;%20v_%7B33%7D%20&amp;%20%5Ccdots%20&amp;%20v_%7B3N%7D%20%5C%5C%0A%5Cvdots%20&amp;%20%5Cvdots%20&amp;%20%5Cvdots%20&amp;%20%5Cvdots%20&amp;%20%5Cddots%20&amp;%20%5Cvdots%20%5C%5C%0Ac_m%20&amp;%20v_%7Bm1%7D%20&amp;%20v_%7Bm2%7D%20&amp;%20v_%7Bm3%7D%20&amp;%20%5Ccdots%20&amp;%20v_%7BmN%7D%0A%5Cend%7Barray%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?m">: 평가자의 수</li>
<li><img src="https://latex.codecogs.com/png.latex?N">: 평가된 항목(unit)의 개수</li>
<li><img src="https://latex.codecogs.com/png.latex?v_%7Bij%7D">: 평가자 <img src="https://latex.codecogs.com/png.latex?c_i">가 특정 단위 <img src="https://latex.codecogs.com/png.latex?u_j">에 대해 부여한 값</li>
<li><img src="https://latex.codecogs.com/png.latex?m_j">: 단위 <img src="https://latex.codecogs.com/png.latex?u_j">에 대해 평가된 개수</li>
</ul>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?m_j">는 특정 단위 <img src="https://latex.codecogs.com/png.latex?u_j">에 대해 평가된 개수이다. 일부 평가자가 특정 단위에 대한 평가를 하지 않을 수도 있기 때문에, <img src="https://latex.codecogs.com/png.latex?m_j">는 평가자가 동일하지 않은 경우 <img src="https://latex.codecogs.com/png.latex?m">보다 작을 수도 있다.</p>
<p>Krippendorff’s Alpha를 계산하려면 평가된 값들이 <strong>서로 비교 가능(pairable)</strong> 해야 하므로, <img src="https://latex.codecogs.com/png.latex?m_j%20%5Cgeq%202">의 조건이 필요하다. 즉, 특정 단위에서 최소 두 명 이상의 평가자가 값을 할당해야 한다. 전체 데이터에서 가능한 쌍의 개수는 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csum_%7Bj=1%7D%5E%7BN%7D%20m_j%20=%20n%20%5Cleq%20mN%0A"></p>
<p>이러한 행렬 표현은 Krippendorff’s Alpha에서 관찰된 불일치 <img src="https://latex.codecogs.com/png.latex?D_o">와 기대되는 불일치 <img src="https://latex.codecogs.com/png.latex?D_e">를 계산하는 기본적인 데이터 구조를 나타낸다.</p>
</section>
<section id="krippendorffs-alpha-공식" class="level2">
<h2 class="anchored" data-anchor-id="krippendorffs-alpha-공식">5.1 Krippendorff’s Alpha 공식</h2>
<p>Krippendorff’s Alpha는 평가자 간의 일치도를 분석하기 위해 관찰된 불일치(<img src="https://latex.codecogs.com/png.latex?D_o">)와 우연히 예상된 불일치(<img src="https://latex.codecogs.com/png.latex?D_e">)를 비교하여 계산된다. 평가자들이 응답할 수 있는 모든 가능한 값들의 집합을 <img src="https://latex.codecogs.com/png.latex?R">이라고 하고, 평가자들이 특정 예제에 대해 내린 응답을 하나의 단위(unit)라고 할 때, Krippendorff’s Alpha는 다음과 같이 정의된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Calpha%20=%201%20-%20%5Cfrac%7BD_o%7D%7BD_e%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?D_o">: 실제로 관찰된 불일치 (Observed Disagreement)</li>
<li><img src="https://latex.codecogs.com/png.latex?D_e">: 우연히 예상된 불일치 (Expected Disagreement by chance)</li>
</ul>
<section id="observed-disagreement-d_o" class="level3">
<h3 class="anchored" data-anchor-id="observed-disagreement-d_o">Observed Disagreement <img src="https://latex.codecogs.com/png.latex?D_o"></h3>
<p><img src="https://latex.codecogs.com/png.latex?D_o">는 다음과 같이 정의된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AD_o%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bc%20%5Cin%20R%7D%20%5Csum_%7Bk%20%5Cin%20R%7D%20%5Cdelta(c,%20k)%20%5Csum_%7Bu%20%5Cin%20U%7D%20m_u%20%5Cfrac%7Bn_%7Bcku%7D%7D%7BP(m_u,%202)%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?%5Cdelta(c,%20k)">: 두 개의 평가 값 <img src="https://latex.codecogs.com/png.latex?c">와 <img src="https://latex.codecogs.com/png.latex?k"> 사이의 차이</li>
<li><img src="https://latex.codecogs.com/png.latex?n">: 총 가능한 쌍(pair)의 개수 - <img src="https://latex.codecogs.com/png.latex?m_u">: 특정 단위 <img src="https://latex.codecogs.com/png.latex?u">에 포함된 평가 수</li>
<li><img src="https://latex.codecogs.com/png.latex?n_%7Bcku%7D">: 단위 <img src="https://latex.codecogs.com/png.latex?u">에서 평가 값 <img src="https://latex.codecogs.com/png.latex?(c,%20k)"> 쌍이 나타난 횟수</li>
<li><img src="https://latex.codecogs.com/png.latex?P">: 순열(permutation)</li>
</ul>
<p>이 식은 평가자들이 특정 단위에서 얼마나 불일치했는지를 개념적으로 가중 평균(weighted average)한 것으로 해석할 수 있다. 여기서 중요한 역할을 하는 <img src="https://latex.codecogs.com/png.latex?%5Cdelta(c,%20k)">는 평가 값 <img src="https://latex.codecogs.com/png.latex?c">와 평가 값 <img src="https://latex.codecogs.com/png.latex?k"> 사이의 차이를 의미하는데, 순위형 데이터의 경우 제곱을 취하여 거리의 크기를 강조하는 방식으로 계산한다. 즉, 두 평가 값이 동일하면 <img src="https://latex.codecogs.com/png.latex?%5Cdelta(c,%20k)%20=%200">이 되고, 평가 값 간 차이가 크면 불일치도가 더 커지는 방식이다.</p>
<ul>
<li>Krippendorff’s Alpha는 데이터의 유형에 따라 다른 <img src="https://latex.codecogs.com/png.latex?%5Cdelta(c,%20k)">의 정의를 사용한다. 이 값을 기반으로 평가자 간의 불일치를 측정하므로, 이 함수가 어떻게 정의가 되느냐에 따라 Alpha <img src="https://latex.codecogs.com/png.latex?%CE%B1">의 값이 달라진다.</li>
</ul>
<p>또한, <img src="https://latex.codecogs.com/png.latex?P(m_u,%202)">는 특정 단위 <img src="https://latex.codecogs.com/png.latex?u">에서 평가된 값들 중에서 2개의 값을 선택하여 순서를 고려한 쌍의 개수를 의미한다. 이는 순열 함수로 표현되며, Krippendorff’s Alpha에서 관찰된 불일치도를 계산할 때 중요한 역할을 한다. 공식은 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP(m_u,%202)%20=%20%5Cfrac%7Bm_u%20(m_u%20-%201)%7D%7B2%7D%0A"></p>
<p>이러한 계산이 필요한 이유는, Krippendorff’s Alpha에서 평가자 간의 불일치를 측정할 때 단순한 개별 평가 값을 비교하는 것이 아니라, 각 단위에서 이루어진 모든 평가 값들 간의 관계를 분석해야 하기 때문이다.</p>
<p>이 식은 평가 값 자체의 범주를 기반으로 전체적인 불일치도를 직접적으로 계산하는 방식이라면, 단위별 불일치도를 먼저 계산한 후 전체 평균을 구하는 방식도 존재한다. <img src="https://latex.codecogs.com/png.latex?D_o">에 대한 단위 중심의 접근법은 다음과 같다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AD_o%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bj=1%7D%5E%7BN%7D%20m_j%20E(%5Cdelta_j)%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?E(%5Cdelta_j)">는 모든 가능한 쌍에 대해 평균적인 거리이며, 아래와 같은 식으로 표현할 수 있다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE(%5Cdelta_j)%20=%20%5Cfrac%7B%20%5Csum_%7Bi%3Ei'%7D%20%5Cdelta(v_%7Bij%7D,%20v_%7Bi'j%7D)%20%7D%7B%5Cbinom%7Bm_j%7D%7B2%7D%7D%0A"></p>
<ul>
<li><img src="https://latex.codecogs.com/png.latex?v_%7Bij%7D">: 평가자 <img src="https://latex.codecogs.com/png.latex?i">가 단위 <img src="https://latex.codecogs.com/png.latex?j">에 대해 부여한 평가 값</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Cdelta(v_%7Bij%7D,%20v_%7Bi'j%7D)">: 두 평가 값 <img src="https://latex.codecogs.com/png.latex?v_%7Bij%7D">와 <img src="https://latex.codecogs.com/png.latex?v_%7Bi'j%7D"> 간의 거리</li>
<li><img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi%3Ei'%7D">: 단위 <img src="https://latex.codecogs.com/png.latex?j">에서 모든 평가자 간의 가능한 쌍을 고려한 합 - <img src="https://latex.codecogs.com/png.latex?%5Cbinom%7Bm_j%7D%7B2%7D">: 단위 <img src="https://latex.codecogs.com/png.latex?j">에서 가능한 모든 평가 쌍의 개수. <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Bm_j(m_j%20-%201)%7D%7B2%7D">로 계산된다.</li>
</ul>
<p>이 수식은 단위 <img src="https://latex.codecogs.com/png.latex?j">에서 평가자들이 부여한 모든 값들을 비교하여 평균적인 불일치도를 구하는 과정이다. 평가자들이 같은 값을 부여했다면 <img src="https://latex.codecogs.com/png.latex?%5Cdelta(v_%7Bij%7D,%20v_%7Bi'j%7D)%20=%200">이 되어 <img src="https://latex.codecogs.com/png.latex?E(%5Cdelta_j)"> 값이 작아지고, 평가 값이 크게 차이 날수록 <img src="https://latex.codecogs.com/png.latex?E(%5Cdelta_j)"> 값이 증가한다.</p>
<p>또한, 만약 모든 단위에서 평가자 수가 일정하다면, <img src="https://latex.codecogs.com/png.latex?D_o">는 전체 평가 단위에서 가능한 모든 평가 쌍에 대한 평균적인 거리로 해석할 수 있다. 이는 일반적으로 평가 행렬에서 대각선에서의 평균적인 거리로 볼 수 있으며, 평가자들이 특정 경향을 가지고 평가했는지 또는 무작위로 평가했는지를 판단하는 중요한 지표가 된다.</p>
</section>
<section id="expected-disagreement-by-chance-d_e" class="level3">
<h3 class="anchored" data-anchor-id="expected-disagreement-by-chance-d_e">Expected Disagreement by chance <img src="https://latex.codecogs.com/png.latex?D_e"></h3>
<p>우연히 예상된 불일치 <img src="https://latex.codecogs.com/png.latex?D_e">는 평가자들이 무작위로 응답했다고 가정할 때 예상되는 불일치도를 의미하는데, 모든 가능한 평가 값 쌍(c, k)의 발생 확률을 이용해 불일치를 추정한다. 이는 Krippendorff’s Alpha에서 평가자 간의 일치도를 측정할 때, 실제 관찰된 불일치도(<img src="https://latex.codecogs.com/png.latex?D_o">)와 비교하는 기준이 된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AD_e%20=%20%5Cfrac%7B1%7D%7BP(n,2)%7D%20%5Csum_%7Bc%20%5Cin%20R%7D%20%5Csum_%7Bk%20%5Cin%20R%7D%20%5Cdelta(c,k)%20P_%7Bck%7D%0A"> - <img src="https://latex.codecogs.com/png.latex?P(n,2)">: 전체 가능한 평가 쌍(pair)의 개수 - <img src="https://latex.codecogs.com/png.latex?%5Cdelta(c,k)">: 두 개의 평가 값 <img src="https://latex.codecogs.com/png.latex?c">와 <img src="https://latex.codecogs.com/png.latex?k"> 사이의 차이 - <img src="https://latex.codecogs.com/png.latex?P_%7Bck%7D">: 특정 평가 값 <img src="https://latex.codecogs.com/png.latex?(c,k)"> 쌍이 발생할 확률</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP_%7Bck%7D%20=%20%5Cbegin%7Bcases%7D%0A%20%20n_c%20n_k%20&amp;%20%5Ctext%7Bif%20%7D%20c%20%5Cneq%20k%20%5C%5C%0A%20%20n_c%20(n_c%20-%201)%20&amp;%20%5Ctext%7Bif%20%7D%20c%20=%20k%0A%5Cend%7Bcases%7D%0A"></p>
<p>이 식에서 <img src="https://latex.codecogs.com/png.latex?n_c">와 <img src="https://latex.codecogs.com/png.latex?n_k">는 각각 평가 값 <img src="https://latex.codecogs.com/png.latex?c">와 <img src="https://latex.codecogs.com/png.latex?k">가 전체 데이터에서 나타난 횟수이다. 만약 <img src="https://latex.codecogs.com/png.latex?c%20%5Cneq%20k">이면, 서로 다른 두 개의 평가 값이 선택될 확률은 <img src="https://latex.codecogs.com/png.latex?n_c%20n_k">로 표현된다. 반면, <img src="https://latex.codecogs.com/png.latex?c%20=%20k">이면 동일한 평가 값이 두 번 선택될 확률은 <img src="https://latex.codecogs.com/png.latex?n_c%20(n_c%20-%201)">로 계산된다. 이러한 확률을 고려하여 평가 값들이 랜덤하게 분포되었을 때 기대되는 불일치도를 구할 수 있다.</p>
<p>즉, <img src="https://latex.codecogs.com/png.latex?D_e">는 평가자들이 일관된 기준 없이 평가한 경우 예상되는 불일치도의 평균적인 크기를 나타낸다.</p>
<p>Krippendorff’s Alpha는 개념적으로 직관적이지만, 계산적으로는 다소 복잡할 수 있다. 그러나 이 지표는 다양한 데이터 유형에 적용할 수 있으며, 평가자의 수가 일정하지 않거나 결측값이 포함된 경우에도 안정적인 신뢰도 분석을 수행할 수 있는 장점이 있다.</p>
</section>
<section id="krippendorffs-alpha-값-해석" class="level3">
<h3 class="anchored" data-anchor-id="krippendorffs-alpha-값-해석">Krippendorff’s Alpha 값 해석</h3>
<p><img src="https://latex.codecogs.com/png.latex?%CE%B1">에 대한 해석은 다음과 같다.</p>
<ul>
<li><strong>α = 1</strong>: 완벽한 평가자 간 일치 (모든 평가자가 동일한 응답)</li>
<li><strong>0.8 ≤ α ≤ 1</strong>: 높은 신뢰도를 의미하며 연구 결과로 활용 가능</li>
<li><strong>0.67 ≤ α &lt; 0.8</strong>: 신뢰할 수 있는 수준이지만 엄격한 연구에서는 보완이 필요함</li>
<li><strong>0 ≤ α &lt; 0.67</strong>: 신뢰도가 낮아 추가적인 평가 기준 수정 또는 평가자 교육이 필요함</li>
<li><strong>α &lt; 0</strong>: 평가자 간 일치도가 우연보다도 낮음 (평가 기준이 모호하거나 데이터에 문제 가능성)</li>
</ul>
</section>
</section>
<section id="example-4" class="level2">
<h2 class="anchored" data-anchor-id="example-4">5.2 Example</h2>
<p>아래는 <code>irrCAC</code> 패키지를 사용하여 Krippendorff’s Alpha를 계산하는 R 코드 예제이다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(irrCAC)</span>
<span id="cb9-2"></span>
<span id="cb9-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 여러 평가자의 병변 평가 데이터</span></span>
<span id="cb9-4">ratings <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(</span>
<span id="cb9-5">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater1 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>),</span>
<span id="cb9-6">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater2 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>),</span>
<span id="cb9-7">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater3 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>),</span>
<span id="cb9-8">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rater4 =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">4</span>, <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">NA</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>, <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)</span>
<span id="cb9-9">)</span>
<span id="cb9-10"></span>
<span id="cb9-11"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Krippendorff’s Alpha 계산 (순위형 데이터)</span></span>
<span id="cb9-12">alpha_result <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">krippen.alpha.raw</span>(ratings, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">weights =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ordinal"</span>)</span>
<span id="cb9-13"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">print</span>(alpha_result)</span></code></pre></div></div>
<p>이 코드는 순위형 데이터를 기반으로 Krippendorff’s Alpha를 계산하는 방법을 보여준다. 평가자 간 일치도를 보다 유연하게 측정할 수 있으며, Fleiss Kappa나 Weighted Cohen’s Kappa보다 데이터 특성에 덜 제한을 받는다는 장점이 있다. 또한, 결측값이 포함된 데이터를 그대로 분석할 수 있다는 점에서 다른 신뢰도 측정법보다 강력한 활용성을 가진다.</p>
<p>Krippendorff’s Alpha는 평가자가 많을수록, 그리고 평가 기준이 명확할수록 신뢰도가 높게 나타난다. 하지만 평가자 간 의견 차이가 크다면 신뢰도 값이 낮아질 수 있으며, 이런 경우 평가 기준을 조정하거나 추가적인 훈련이 필요할 수 있다.</p>
</section>
</section>
<section id="마치며" class="level1">
<h1>마치며</h1>
<p>마지막으로 유의할 점은 Kappa가 두 평가자 사이에 평가 결과가 얼마나 비슷한지, 즉 평가자들이 서로 얼마나 일치하는지만 측정할 수 있다는 것이다. 이것을 ’신뢰도(Reliability)’라고 한다. 반면, 평가자들이 실제로 올바른 평가를 하고 있는지 여부, 즉 ’타당도(Validity)’에 대해서는 알려주지 못한다. 평가가 옳고 정확한지 판단하는 것은 타당도의 영역이며, Kappa는 오직 신뢰도를 측정할 때만 사용할 수 있다는 점을 기억하자.</p>


</section>

<div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{kang2025,
  author = {Kang, YeJi},
  title = {Kappa {분석} {이해하기}},
  date = {2025-03-18},
  url = {https://blog.zarathu.com/posts/2025-03-18-Kappa/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-kang2025" class="csl-entry quarto-appendix-citeas">
Kang, YeJi. 2025. <span>“Kappa 분석 이해하기.”</span> March 18, 2025. <a href="https://blog.zarathu.com/posts/2025-03-18-Kappa/">https://blog.zarathu.com/posts/2025-03-18-Kappa/</a>.
</div></div></section></div> ]]></description>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-03-18-Kappa/</guid>
  <pubDate>Tue, 18 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-03-18-Kappa/img/Kappa_uc_lc.svg.png" medium="image" type="image/png" height="96" width="144"/>
</item>
<item>
  <title>quarto 의 기초</title>
  <dc:creator>YuJeong Yoon</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-03-13-Quarto/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><p><a href="https://quarto.org/"><strong>Quarto</strong></a><strong>를 이용해</strong> <strong>R</strong> 코드와 분석 결과가 포함된 문서를 작성하는 방법을 강의할 예정입니다. 강의 내용을 미리 공유합니다.</p>
<section id="시작하기-전에" class="level2"><h2 class="anchored" data-anchor-id="시작하기-전에">시작하기 전에</h2>
<p><img src="https://bioinformatics.ccr.cancer.gov/docs/btep-coding-club/CC2024/Quarto/images/quarto_process.png" class="img-fluid" alt="Quarto"><sup>1</sup></p>
<p><a href="https://quarto.org/"><strong>Quarto</strong></a><strong>는 Markdown을 기반으로 한 문서 작성 도구로,</strong> <a href="https://quarto.org/docs/computations/python.html">Python</a>,&nbsp;<strong>R</strong> ,&nbsp;<a href="https://quarto.org/docs/computations/julia.html">Julia</a>, and&nbsp;<a href="https://quarto.org/docs/computations/ojs.html">Observable</a> 등 다양한 언어로 코드실행, 분석, 시각화를 포함한 컨텐츠를 만드는 툴이며 크게 3가지 활용법이 있다.</p>
<ol type="1">
<li><p>문서(<code>pdf</code>, <code>html</code>, <code>docx</code>): 글쓰기, 분석 결과, 참고문헌 등 논문의 모든 작업을 <a href="https://quarto.org/"><strong>Quarto</strong></a>으로 수행한다.</p></li>
<li><p>프리젠테이션(<code>pdf</code>, <code>html</code>, <code>pptx</code>): <a href="https://www.r-project.org/"><strong>R</strong></a> 코드나 분석결과가 포함된 프리젠테이션을 만든다. <a href="https://rmarkdown.rstudio.com/lesson-11.html">기본 템플릿</a><sup>2</sup> 외에 <a href="https://github.com/yihui/xaringan"><strong>xaringan</strong></a><sup>3</sup> 패키지가 최근 인기를 끌고 있다.</p></li>
<li><p>웹(<code>html</code>): 웹사이트나 블로그를 만든다. <a href="https://github.com/rstudio/blogdown"><strong>blogdown</strong></a><sup>4</sup> 이나 <a href="https://rstudio.github.io/distill/"><strong>distill</strong></a><sup>5</sup> 패키지가 대표적이다. 이 글의 블로그도 <a href="https://rstudio.github.io/distill/"><strong>distill</strong></a>로 만들었으며, 과거 차라투 홈페이지는 <a href="https://github.com/rstudio/blogdown"><strong>blogdown</strong></a>을 이용하였다.</p></li>
</ol>
<p>본 강의는 1의 가장 기초에 해당하는 강의로 간단한 문서를 작성하는 것을 목표로 한다. <code>pdf</code> 문서를 만들기 위해서는 추가로 <a href="https://www.latex-project.org/"><strong>LaTeX</strong></a> 문서작성 프로그램인 <a href="http://www.ktug.org/xe/?mid=Install"><strong>Tex Live</strong></a>를 설치해야 하며 본 강의에서는 생략한다.</p>
</section><section id="qmd-문서-시작하기" class="level2"><h2 class="anchored" data-anchor-id="qmd-문서-시작하기">.qmd 문서 시작하기</h2>
<p><a href="https://quarto.org/"><strong>Quarto</strong></a>는 <strong>qmd</strong> 파일로 작성되며 <a href="https://github.com/quarto-dev/quarto-r"><strong>Quarto</strong></a><sup>6</sup> 패키지를 설치한 후, <a href="https://www.rstudio.com/"><strong>Rstudio</strong></a>에서 <strong>File</strong> <img src="https://latex.codecogs.com/png.latex?%5Crightarrow"> New File <img src="https://latex.codecogs.com/png.latex?%5Crightarrow"> Quarto Document… 의 순서로 클릭하여 시작할 수 있다.</p>
<p><img src="https://user-images.githubusercontent.com/31009952/163181083-9fc6e19a-dcc4-44d3-97cb-9c9a7bf2bcbe.png" class="img-fluid" alt="Rstudio File 메뉴"> Rstudio File 메뉴<sup>7</sup></p>
<p><img src="https://quarto.org/docs/tools/images/new-quarto-doc.png" class="img-fluid" alt="Quarto 시작 메뉴"> Quarto 시작 메뉴<sup>8</sup></p>
<p>문서의 제목과 저자 이름을 적은 후 파일 형태를 아무거나 고르면(나중에도 쉽게 수정 가능)확장자가 <code>qmd</code>인 문서가 생성될 것이다.</p>
<p>다음은 각각 <code>html</code>, <code>pdf</code>, <code>docx</code>로 생성된 문서이다.</p>
<div class="cell">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/html.png" class="img-fluid figure-img" width="271"></p>
<figcaption>html 문서</figcaption></figure>
</div>
</div>
</div>
<div class="cell">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/pdf.png" class="img-fluid figure-img" width="366"></p>
<figcaption>pdf 문서</figcaption></figure>
</div>
</div>
</div>
<div class="cell">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/docx.png" class="img-fluid figure-img" width="365"></p>
<figcaption>word 문서</figcaption></figure>
</div>
</div>
</div>
<p>생각보다 간단하지 않은가? 이제 본격적으로 <code>qmd</code> 파일의 내용을 살펴보면서 어떻게 글과 코드를 작성하는지 알아보자. <code>qmd</code>는 크게 제목을 적는 <strong>YAML Header</strong>, 글을 쓰는 <strong>Markdown Text</strong>와 코드를 적는 <strong>Code Chunk</strong>로 나눌 수 있다.</p>
<div class="cell">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/qmd.png" class="img-fluid figure-img" width="685"></p>
<figcaption>qmd 구성 예시</figcaption></figure>
</div>
</div>
</div>
</section><section id="yaml-header" class="level2"><h2 class="anchored" data-anchor-id="yaml-header">YAML Header</h2>
<p><code>YAML</code>은 <strong>Y</strong>AML <strong>A</strong>in’t <strong>M</strong>arkup <strong>L</strong>anguage의 재귀형식의 이름을 갖고 있는 언어로 가독성에 초점을 두고 개발되었다. <a href="https://quarto.org/"><strong>Quarto</strong></a>은 <code>qmd</code>의 시작 부분에 문서 형식을 설정하는 용도로 이 포맷을 이용한다. 다음은 기초 정보만 포함된 <code>YAML</code>이다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode yaml code-with-copy"><code class="sourceCode yaml"><span id="cb1-1"><span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">---</span></span>
<span id="cb1-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">title</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> </span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"My Document"</span></span>
<span id="cb1-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">format</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb1-4"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">  </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">html</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb1-5"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">toc</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> </span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">true</span></span>
<span id="cb1-6"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">number-sections</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> </span><span class="ch" style="color: #20794D;
background-color: null;
font-style: inherit;">true</span></span>
<span id="cb1-7"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">css</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> styles.css</span></span>
<span id="cb1-8"><span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">---</span></span></code></pre></div></div>
<p><code>YAML</code> 에 기본적으로 title, author, date 등을 작성할 수 있고, format을 통해 출력 형식 (e.g html, pdf, docx…) 과 각 형식에 맞는 다양한 옵션을 설정하여 문서를 꾸밀 수 있다. Table of Contents, Layout, Fonts 등 다양한 옵션의 설정이 가능하고, 여기서는 Toc 옵션에 대해 살펴볼 것이다. 자세한 format option은 <a href="https://quarto.org/docs/reference/formats/html.html#figures">Quarto reference</a>를 참고하기 바란다.</p>
<section id="toc" class="level3"><h3 class="anchored" data-anchor-id="toc">toc</h3>
<p>Quarto 문서(.qmd)에서 ##을 사용해 제목을 작성하면, 자동으로 목차(TOC)에 포함된다. 하위 목차를 추가하려면 ###, ####처럼 #의 개수를 늘려 계층 구조를 만들 수 있다.</p>
<p><code>toc</code> 옵션에는 toc-depth, toc-location, toc-title, toc-expand 가 있다. 이 문서의 yaml 부분의 toc 옵션을 살펴보면 다음과 같이 설정되어있다.</p>
<pre><code>---
title: "quarto 의 기초"
author: .
editor: visual
format: 
  html: 
    toc-depth: 3
    toc-expand: true
    toc-location: left
    toc-title: "Quarto 알아보기"
---</code></pre>
</section></section><section id="markdown-글쓰기" class="level2"><h2 class="anchored" data-anchor-id="markdown-글쓰기">Markdown 글쓰기</h2>
<p>Markdown 은 이름에서 알 수 있듯이 <a href="https://gist.github.com/ihoneymon/652be052a0727ad59601"><strong>마크다운(Markdown)</strong></a> 을 기반으로 만들어졌다. <a href="https://gist.github.com/ihoneymon/652be052a0727ad59601"><strong>마크다운</strong></a>은 문법이 매우 간단한 것이 특징으로 <a href="https://github.com"><strong>깃허브</strong></a>의 <strong><code>README.md</code></strong>가 대표적인 마크다운 문서이다. Quarto 는 Pandoc markdown 을 바탕으로 하며, <a href="https://quarto.org/docs/authoring/markdown-basics.html">quarto guide</a><sup>9</sup>에 흔히 쓰는 문법이 정리되어 있다.</p>
<p>2 가지만 따로 살펴보겠다.</p>
<section id="inline-r-code" class="level3"><h3 class="anchored" data-anchor-id="inline-r-code">Inline R code</h3>
<p>문장 안에 분석 결과값을 적을 때, 분석이 바뀔 때마다 바뀐 숫자를 직접 수정해야 한다. 그러나 숫자 대신 <code>`r &lt;코드&gt;`</code> 꼴로 <a href="https://www.r-project.org/"><strong>R</strong></a> 코드를 넣는다면 재분석시 그 숫자를 자동으로 업데이트 시킬 수 있다.</p>
<pre class="rmd"><code>There were  `r nrow(cars)` cars studied</code></pre>
<blockquote class="blockquote">
<p>There were 50 cars studied</p>
</blockquote>
</section><section id="수식" class="level3"><h3 class="anchored" data-anchor-id="수식">수식</h3>
<p><a href="https://www.latex-project.org/"><strong>LaTeX</strong></a> 문법을 사용하며 <code>hwp</code> 문서의 수식 편집과 비슷하다. inline 삽입은 <code>$...$</code>, 새로운 줄은 <code>$$...$$</code> 안에 식을 적으면 된다.</p>
<pre class="rmd"><code>This summation expression $\sum_{i=1}^n X_i$ appears inline.</code></pre>
<blockquote class="blockquote">
<p>This summation expression <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bi=1%7D%5En%20X_i"> appears inline.</p>
</blockquote>
<pre><code>$$
\sigma = \sqrt{ \frac{1}{N} \sum_{i=1}^N (x_i -\mu)^2}
$$</code></pre>
<p><img src="https://latex.codecogs.com/png.latex?%5Csigma%20=%20%5Csqrt%7B%20%5Cfrac%7B1%7D%7BN%7D%20%5Csum_%7Bi=1%7D%5EN%20(x_i%20-%5Cmu)%5E2%7D"></p>
<p>수식 전반은 <a href="https://www.latex-tutorial.com/tutorials/amsmath/">LaTeX math and equations</a><sup>10</sup>을 참고하기 바란다.</p>
</section></section><section id="r-chunk" class="level2"><h2 class="anchored" data-anchor-id="r-chunk">R chunk</h2>
<p>Quarto 에서는 내장된 <a href="https://github.com/yihui/knitr">knitr 패키지</a>을 이용하여 R에서 작성한 코드를 실행하고 그 결과를 실시간으로 출력하여 문서에 삽입할 수 있다.</p>
<section id="r-chunk-생성하기" class="level3"><h3 class="anchored" data-anchor-id="r-chunk-생성하기">R chunk 생성하기</h3>
<p>R chunk 를 생성하는 방법은 위의 단추를 통해 생성하거나 <img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/rchunk.png" class="img-fluid" alt="r chunk"></p>
<p>혹은 다음과 같이 직접 타이핑하여 생성도 가능하다. <img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/chunk.png" class="img-fluid" alt="chunk"></p>
</section><section id="r-chunk-옵션" class="level3"><h3 class="anchored" data-anchor-id="r-chunk-옵션">R chunk 옵션</h3>
<p><strong>.qmd</strong> 문서에서 <a href="https://www.r-project.org/"><strong>R</strong></a> 코드가 들어가는 방식은 4가지이다.</p>
<ol type="1">
<li><p>몰래 실행. 코드와 결과는 다 숨긴다</p></li>
<li><p>실행. 코드와 결과를 모두 보여준다. - default</p></li>
<li><p>실행. 코드는 숨기고 결과만 보여준다.</p></li>
<li><p>실행하지 않음. 코드 보여주기만 한다.</p></li>
</ol>
<p><code>include</code>, <code>echo</code>, <code>eval</code> 3가지 옵션으로 지정한다. - <code>eval=F</code> : 코드를 실행하지 않는다. - <code>echo=F</code> : 코드를 보여주지 않는다. - <code>include=F</code> : 실행 결과를 보여주지 않는다.</p>
<p>코드 청크의 옵션은 YAML 에서 지정하여 문서 전체에 적용되게 할 수 있고, 각각 R 청크마다 <code>#|</code> 을 쳐서 각각 옵션을 변경할 수도 있다.</p>
<section id="최초-설정" class="level4"><h4 class="anchored" data-anchor-id="최초-설정">최초 설정</h4>
<p>문서를 처음 생성 시 옵션을 따로 지정하지 않으면 다음의 값으로 실행된다.</p>
<pre><code>include = TRUE 
echo = TRUE 
eval = TRUE </code></pre>
<p>코드를 실행하고, 코드와 결과물 모두 문서에 보여준다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/print.html">print</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Hello world"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>[1] "Hello world"</code></pre>
</div>
</div>
<p>이를 잘 활용하여 R내에서 문서를 완성할 수 있다.</p>
<p>생존곡선을 그릴 때를 생각해보자, 생존 곡선을 그릴 때는 먼저 survfit 함수를 통해 생존확률을 구해야 한다.</p>
<p>이 때 survfit 함수는 결과를 보이지 않아도 되므로</p>
</section><section id="실행하고-결과를-보이지-않기" class="level4"><h4 class="anchored" data-anchor-id="실행하고-결과를-보이지-않기">실행하고 결과를 보이지 않기</h4>
<p>다음은 이 html을 생성할 때 쓴 quarto 문서의 캡처본으로 <img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/surv.png" class="img-fluid"></p>
<p>현재 강의 화면인 html 에는 코드 및 결과를 보이지 않는다.</p>
</section><section id="실행하고-결과를-보여주기" class="level4"><h4 class="anchored" data-anchor-id="실행하고-결과를-보여주기">실행하고 결과를 보여주기</h4>
<p>그러나 실행은 되었기 때문에 다음의 코드에서 fit3 에 대한 ggsurvplot 함수를 적용할 수 있었고 코드 및 실행 후 결과는 다음과 같다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ggsurv</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ggsurvplot</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">fit3</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">colon</span>,</span>
<span>  fun <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"cumhaz"</span>, conf.int <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span>,</span>
<span>  risk.table <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">F</span>, risk.table.col<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"strata"</span>,</span>
<span>  ggtheme <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">theme_bw</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ggsurv</span></span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/index_files/figure-html/unnamed-chunk-5-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
<p>이렇게 시행하여 echo = False 옵션까지 적용하면 quarto 내에서 분석을 시행하고 그에 대한 문서작성을 한번에 할 수 있다.</p>
<p>이외에도 코드 청크에 다음과 같은 옵션을 적용 가능하다.</p>
<ul>
<li>
<code>message=F</code> - 실행 때 나오는 메세지를 보여주지 않는다.</li>
<li>
<code>warning=F</code> - 실행 때 나오는 경고를 보여주지 않는다.</li>
<li>
<code>error=T</code> - 에러가 있어도 실행하고 에러코드를 보여준다.</li>
<li>
<code>fig.height = 7</code> - 그림 높이, <a href="https://www.r-project.org/"><strong>R</strong></a>로 그린 그림에만 해당한다.</li>
<li>
<code>fig.width = 7</code> - 그림 너비, <a href="https://www.r-project.org/"><strong>R</strong></a>로 그린 그림에만 해당한다.</li>
<li>
<code>fig.align = 'center'</code> - 그림 위치, <a href="https://www.r-project.org/"><strong>R</strong></a>로 그린 그림에만 해당한다.</li>
</ul>
<p>r chunk 에 적용할 수 있는 전체 옵션은 <code>knitr::opts_chunk$get</code> 함수로 확인할 수 있다. `</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb10" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">knitr</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/knitr/man/opts_chunk.html">opts_chunk</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">get</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>$eval
[1] TRUE

$echo
[1] TRUE

$results
[1] "markup"

$tidy
[1] FALSE

$tidy.opts
NULL

$collapse
[1] FALSE

$prompt
[1] FALSE

$comment
[1] NA

$highlight
[1] TRUE

$size
[1] "normalsize"

$background
[1] "#F7F7F7"

$strip.white
[1] TRUE

$cache
[1] FALSE

$cache.path
[1] "index_cache/html/"

$cache.vars
NULL

$cache.lazy
[1] TRUE

$dependson
NULL

$autodep
[1] FALSE

$cache.rebuild
[1] FALSE

$fig.keep
[1] "high"

$fig.show
[1] "asis"

$fig.align
[1] "default"

$fig.path
[1] "index_files/figure-html/"

$dev
[1] "png"

$dev.args
NULL

$dpi
[1] 96

$fig.ext
NULL

$fig.width
[1] 7

$fig.height
[1] 5

$fig.env
[1] "figure"

$fig.cap
NULL

$fig.scap
NULL

$fig.lp
[1] "fig:"

$fig.subcap
NULL

$fig.pos
[1] ""

$out.width
NULL

$out.height
NULL

$out.extra
NULL

$fig.retina
[1] 2

$external
[1] TRUE

$sanitize
[1] FALSE

$interval
[1] 1

$aniopts
[1] "controls,loop"

$warning
[1] FALSE

$error
[1] FALSE

$message
[1] FALSE

$render
NULL

$ref.label
NULL

$child
NULL

$engine
[1] "R"

$split
[1] FALSE

$include
[1] TRUE

$purl
[1] TRUE

$fig.asp
NULL

$fenced.echo
[1] FALSE

$ft.shadow
[1] FALSE</code></pre>
</div>
</div>
<p>다음은 필자가 논문을 quarto로 쓸 때 흔히 쓰는 디폴트 옵션이다. <img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/de.png" class="img-fluid"></p>
</section></section></section><section id="figures" class="level2"><h2 class="anchored" data-anchor-id="figures">Figures</h2>
<p>.qmd 문서에 그림이 들어가는 방법은 2가지가 있다.</p>
<ol type="1">
<li><p><a href="https://www.r-project.org/"><strong>R</strong></a> 코드로 생성 : <code>plot</code> 함수, <code>ggplot2</code> 패키지 등</p></li>
<li><p>외부 그림 삽입</p></li>
</ol>
<p>앞서도 언급했듯이 주의할 점은 그림이 만들어지는 방법에 따라 <strong>서로 다른 옵션</strong>이 적용된다는 것이다. 일단 전자부터 살펴보자.</p>
<section id="figures-with-r" class="level3"><h3 class="anchored" data-anchor-id="figures-with-r">Figures with <a href="https://www.r-project.org/"><strong>R</strong></a>
</h3>
<p><a href="https://www.r-project.org/"><strong>R</strong></a> 코드에서 자체적으로 만든 그림은 전부 chunk 옵션의 지배를 받아 간단하다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb12" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#|fig-cap: "scatterplot: cars"</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#|fig-width: 8</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#|fig-height: 6</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/graphics/plot.default.html">plot</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cars</span>, pch <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">18</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div>
<figure class="figure"><p><img src="https://blog.zarathu.com/posts/2025-03-13-Quarto/index_files/figure-html/unnamed-chunk-7-1.png" class="img-fluid figure-img" width="672"></p>
</figure>
</div>
</div>
</div>
</section><section id="external-figures" class="level3"><h3 class="anchored" data-anchor-id="external-figures">External figures</h3>
<p>외부 그림은 <a href="https://www.r-project.org/"><strong>R</strong></a> 코드로도 삽입할 수 있고 <a href="https://gist.github.com/ihoneymon/652be052a0727ad59601"><strong>마크다운</strong></a> 문법을 쓸 수도 있는데, <strong>어떤 방법을 쓰느냐에 따라 다른 옵션을 적용</strong>받는다는 것을 주의해야 한다. <a href="https://www.r-project.org/"><strong>R</strong></a>에서는 <code><a href="https://rdrr.io/pkg/knitr/man/include_graphics.html">knitr::include_graphics</a></code> 함수를 이용하여 그림을 넣을 수 있고 이 때는 chunk 내부의 옵션이 적용된다.</p>
<div class="cell" data-layout-align="center">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb13" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://yihui.org/knitr/">knitr</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/knitr/man/include_graphics.html">include_graphics</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"https://www.tidyverse.org/images/tidyverse-default.png"</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://www.tidyverse.org/images/tidyverse-default.png" class="img-fluid figure-img"></p>
<figcaption>tidyverse logo</figcaption></figure>
</div>
</div>
</div>
<p>같은 그림을 chunk없이 바로 <a href="https://gist.github.com/ihoneymon/652be052a0727ad59601"><strong>마크다운</strong></a>에서 삽입할 수도 있다. 이 때는 <code>YAML</code>의 옵션이 적용된다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode md code-with-copy"><code class="sourceCode markdown"><span id="cb14-1"><span class="al" style="color: #AD0000;
background-color: null;
font-style: inherit;">![tidyverse logo](https://www.tidyverse.org/images/tidyverse-default.png)</span>{ width=50% }</span></code></pre></div></div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://www.tidyverse.org/images/tidyverse-default.png" class="img-fluid figure-img" style="width:50.0%"></p>
<figcaption>tidyverse logo</figcaption></figure>
</div>
<p><code>{ width=50% }</code> 는 그림의 크기를 조절하는 옵션이며 <a href="https://www.r-project.org/"><strong>R</strong></a> chunk에서도 같은 옵션 <code>out.width="50%"</code>이 있다. 위치를 가운데로 조절하려면 <code>&lt;center&gt;...&lt;/center&gt;</code> 를 포함시키자.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb15" style="background: #f1f3f5;"><pre class="sourceCode md code-with-copy"><code class="sourceCode markdown"><span id="cb15-1"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">center</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span>
<span id="cb15-2"><span class="al" style="color: #AD0000;
background-color: null;
font-style: inherit;">![tidyverse logo](https://www.tidyverse.org/images/tidyverse-default.png)</span>{ width=50% }</span>
<span id="cb15-3"><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;/</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">center</span><span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;</span></span></code></pre></div></div>
<center>
<div class="quarto-figure quarto-figure-center">
<figure class="figure"><p><img src="https://www.tidyverse.org/images/tidyverse-default.png" class="img-fluid figure-img" style="width:50.0%"></p>
<figcaption>tidyverse logo</figcaption></figure>
</div>
</center>
<p>개인적으로는 외부 이미지도 chunk 내부에서 읽는 것을 추천한다. chunk 내부의 옵션들이 <a href="https://gist.github.com/ihoneymon/652be052a0727ad59601"><strong>마크다운</strong></a>의 그것보다 훨씬 체계적이고 쉬운 느낌이다.</p>
</section></section><section id="tables" class="level2"><h2 class="anchored" data-anchor-id="tables">Tables</h2>
<p>논문을 쓸 때 가장 귀찮은 부분 중 하나가 분석 결과를 테이블로 만드는 것으로, <code><a href="https://rdrr.io/pkg/knitr/man/kable.html">knitr::kable()</a></code> 함수를 쓰면 문서 형태에 상관없이 Rmd에서 바로 테이블을 만들 수 있다. 아래는 데이터를 살펴보는 가장 간단한 예시이다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb16" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#|label: "tables-mtcars"</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">knitr</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/knitr/man/kable.html">kable</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">iris</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span>, caption <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">'A caption'</span>, row.names <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<caption>A caption</caption>
<thead><tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: right;">Sepal.Length</th>
<th style="text-align: right;">Sepal.Width</th>
<th style="text-align: right;">Petal.Length</th>
<th style="text-align: right;">Petal.Width</th>
<th style="text-align: left;">Species</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">1</td>
<td style="text-align: right;">5.1</td>
<td style="text-align: right;">3.5</td>
<td style="text-align: right;">1.4</td>
<td style="text-align: right;">0.2</td>
<td style="text-align: left;">setosa</td>
</tr>
<tr class="even">
<td style="text-align: left;">2</td>
<td style="text-align: right;">4.9</td>
<td style="text-align: right;">3.0</td>
<td style="text-align: right;">1.4</td>
<td style="text-align: right;">0.2</td>
<td style="text-align: left;">setosa</td>
</tr>
<tr class="odd">
<td style="text-align: left;">3</td>
<td style="text-align: right;">4.7</td>
<td style="text-align: right;">3.2</td>
<td style="text-align: right;">1.3</td>
<td style="text-align: right;">0.2</td>
<td style="text-align: left;">setosa</td>
</tr>
<tr class="even">
<td style="text-align: left;">4</td>
<td style="text-align: right;">4.6</td>
<td style="text-align: right;">3.1</td>
<td style="text-align: right;">1.5</td>
<td style="text-align: right;">0.2</td>
<td style="text-align: left;">setosa</td>
</tr>
<tr class="odd">
<td style="text-align: left;">5</td>
<td style="text-align: right;">5.0</td>
<td style="text-align: right;">3.6</td>
<td style="text-align: right;">1.4</td>
<td style="text-align: right;">0.2</td>
<td style="text-align: left;">setosa</td>
</tr>
</tbody>
</table>
</div>
</div>
<p><strong>epiDisplay</strong> 패키지의 <code>regress.display</code>, <code>logistic.display</code> 함수를 활용하면 회귀분석의 결과를 바로 테이블로 나타낼 수 있다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb17" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#|label: "regtable"</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mtcars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">as.factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mtcars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mtcars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cyl</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/factor.html">as.factor</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mtcars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cyl</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mpg</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">disp</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">vs</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">cyl</span>, data <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">mtcars</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model.display</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">epiDisplay</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">regress.display</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model</span>, crude <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span>, crude.p.value <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">T</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model.table</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model.display</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">[</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/colnames.html">rownames</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model.display</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">table</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">!=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">""</span>, <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">]</span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/pkg/knitr/man/kable.html">kable</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model.table</span>, caption <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">model.display</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">first.line</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div class="cell-output-display">
<table class="caption-top table table-sm table-striped small">
<caption>Linear regression predicting mpg</caption>
<colgroup>
<col style="width: 19%">
<col style="width: 23%">
<col style="width: 14%">
<col style="width: 21%">
<col style="width: 10%">
<col style="width: 10%">
</colgroup>
<thead><tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: left;">crude coeff.(95%CI)</th>
<th style="text-align: left;">crude P value</th>
<th style="text-align: left;">adj. coeff.(95%CI)</th>
<th style="text-align: left;">P(t-test)</th>
<th style="text-align: left;">P(F-test)</th>
</tr></thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">disp (cont. var.)</td>
<td style="text-align: left;">-0.04 (-0.05,-0.03)</td>
<td style="text-align: left;">&lt; 0.001</td>
<td style="text-align: left;">-0.03 (-0.05,0)</td>
<td style="text-align: left;">0.019</td>
<td style="text-align: left;">&lt; 0.001</td>
</tr>
<tr class="even">
<td style="text-align: left;">vs: 1 vs 0</td>
<td style="text-align: left;">7.94 (4.6,11.28)</td>
<td style="text-align: left;">&lt; 0.001</td>
<td style="text-align: left;">0.04 (-3.81,3.89)</td>
<td style="text-align: left;">0.984</td>
<td style="text-align: left;">0.334</td>
</tr>
<tr class="odd">
<td style="text-align: left;">cyl: ref.=4</td>
<td style="text-align: left;"></td>
<td style="text-align: left;"></td>
<td style="text-align: left;"></td>
<td style="text-align: left;"></td>
<td style="text-align: left;">0.041</td>
</tr>
<tr class="even">
<td style="text-align: left;">6</td>
<td style="text-align: left;">-6.92 (-10.11,-3.73)</td>
<td style="text-align: left;">&lt; 0.001</td>
<td style="text-align: left;">-4.77 (-8.56,-0.98)</td>
<td style="text-align: left;">0.016</td>
<td style="text-align: left;"></td>
</tr>
<tr class="odd">
<td style="text-align: left;">8</td>
<td style="text-align: left;">-11.56 (-14.22,-8.91)</td>
<td style="text-align: left;">&lt; 0.001</td>
<td style="text-align: left;">-4.75 (-12.19,2.7)</td>
<td style="text-align: left;">0.202</td>
<td style="text-align: left;"></td>
</tr>
</tbody>
</table>
</div>
</div>
<p>테이블을 좀 더 다듬으려면 <a href="https://github.com/haozhu233/kableExtra"><strong>kableExtra</strong></a> 패키지가 필요하며, 자세한 내용은 <a href="https://cran.r-project.org/web/packages/kableExtra/vignettes/awesome_table_in_html.html"><strong>cran 설명서</strong></a><sup>11</sup>를 참고하기 바란다. <code>html</code> 문서의 경우 <code><a href="https://rdrr.io/pkg/knitr/man/kable.html">kable()</a></code>외에도 다양한 함수들을 이용할 수 있는데, 대표적인 것이 <code><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">rmarkdown::paged_table()</a></code> 함수와 <a href="https://github.com/rstudio/DT"><strong>DT</strong></a> 패키지이다. 전자는 아래와 같이 <code>YAML</code>에서 테이블 보기의 기본 옵션으로 설정할 수도 있다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb18" style="background: #f1f3f5;"><pre class="sourceCode yaml code-with-copy"><code class="sourceCode yaml"><span id="cb18-1"><span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">---</span></span>
<span id="cb18-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">title</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> </span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Motor Trend Car Road Tests"</span></span>
<span id="cb18-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">output</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb18-4"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">  </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">html_document</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span></span>
<span id="cb18-5"><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">    </span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">df_print</span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">:</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;"> paged</span></span>
<span id="cb18-6"><span class="pp" style="color: #AD0000;
background-color: null;
font-style: inherit;">---</span></span></code></pre></div></div>
<p><a href="https://github.com/rstudio/DT"><strong>DT</strong></a> 패키지에 대한 설명은 <a href="https://rstudio.github.io/DT/">Rstudio DT 홈페이지</a><sup>12</sup>를 참고하기 바란다.</p>
</section><section id="마치며" class="level2"><h2 class="anchored" data-anchor-id="마치며">마치며</h2>
<p>본 강의를 통해 <a href="https://quarto.org/"><strong>Quarto</strong></a>으로 기본적인 문서를 만드는 법을 알아보았다. 본 강의에서는 시간 관계상 참고문헌 다는 법을 언급하지 않았는데 궁금하다면 <a href="https://pandoc.org/MANUAL.html#citations">Bibliographies and Citations</a><sup>13</sup>을 참고하자.</p>
<p>이 내용까지 숙지한다면 <a href="https://quarto.org/"><strong>Quarto</strong></a>으로 논문을 쓸 준비가 된 것이다. <a href="https://quarto.org/"><strong>Quarto</strong></a>에 대한 전반적인 내용은 아래의 <a href="https://rstudio.github.io/cheatsheets/quarto.pdf">Quarto Cheet Sheet</a><sup>14</sup>에 잘 요약되어 있으니 그때그떄 확인하면 좋다.</p>


</section><div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>
<ol>
<li id="fn1"><p>https://bioinformatics.ccr.cancer.gov/docs/btep-coding-club/CC2024/Quarto/GettingStarted_with_Quarto_orig.html↩︎</p></li>
<li id="fn2"><p>https://rmarkdown.rstudio.com/lesson-11.html↩︎</p></li>
<li id="fn3"><p>https://github.com/yihui/xaringan↩︎</p></li>
<li id="fn4"><p>https://github.com/rstudio/blogdown↩︎</p></li>
<li id="fn5"><p>https://rstudio.github.io/distill/↩︎</p></li>
<li id="fn6"><p>https://github.com/quarto-dev/quarto-r↩︎</p></li>
<li id="fn7"><p>https://github.com/rstudio/rstudio/issues/10966↩︎</p></li>
<li id="fn8"><p>https://github.com/quarto-dev/quarto-r↩︎</p></li>
<li id="fn9"><p>https://quarto.org/docs/authoring/markdown-basics.html↩︎</p></li>
<li id="fn10"><p>https://www.latex-tutorial.com/tutorials/amsmath/↩︎</p></li>
<li id="fn11"><p>https://cran.r-project.org/web/packages/kableExtra/vignettes/awesome_table_in_html.html↩︎</p></li>
<li id="fn12"><p>https://rstudio.github.io/DT/↩︎</p></li>
<li id="fn13"><p>https://pandoc.org/MANUAL.html#citations↩︎</p></li>
<li id="fn14"><p>https://rstudio.github.io/cheatsheets/quarto.pdf↩︎</p></li>
</ol></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{yoon2025,
  author = {Yoon, YuJeong},
  title = {Quarto {의} {기초}},
  date = {2025-03-13},
  url = {https://blog.zarathu.com/posts/2025-03-13-Quarto/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-yoon2025" class="csl-entry quarto-appendix-citeas">
Yoon, YuJeong. 2025. <span>“Quarto 의 기초.”</span> March 13, 2025. <a href="https://blog.zarathu.com/posts/2025-03-13-Quarto/">https://blog.zarathu.com/posts/2025-03-13-Quarto/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <guid>https://blog.zarathu.com/posts/2025-03-13-Quarto/</guid>
  <pubDate>Thu, 13 Mar 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-03-13-Quarto/logo.png" medium="image" type="image/png" height="35" width="144"/>
</item>
<item>
  <title>Exploring Regression Models for Regression Analysis (1): Regression Analysis, Linear Regression, HC Standard Errors</title>
  <dc:creator>Lee Seungjun</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-02-28-reg1/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="들어가며" class="level2"><h2 class="anchored" data-anchor-id="들어가며">들어가며</h2>
<p>차라투 블로그 “Exploring Regression Models for Regression Analysis”에서는 세(네) 장에 걸쳐서 통계에서의 Regression Analysis를 위한 여러 Regression Model들을 수학적으로 깊게 탐구합니다. 1장에서는 Regression Analysis의 개념, (simple, multiple or general) linear regression에 대한 개념 및 Analysis 과정에 대한 소개로 시작해서 data의 특성 중 하나인 Homo, Heteroskedasticity의 개념, Heteroskedasticity로 추정되는 경우 linear regression 모델의 robust한 (co)variance를 구하는 방법 중 하나이자 필수로 고려해야 하는 Heteroskedasticity-consistent standard errors (heteroskedasticity-robust standard errors)와 Wild Bootstrap를 살펴볼 것입니다. 2장에서는 1장에서 다룬 기본 linear regression에서 link function을 도입하여 regression의 개념을 outcome of single yes/no, outcome of single K-way, count 등 non-normal한 종속변수로 확장한 Generalized linear model의 개념을 Exponential Family, Link Function와 같은 핵심 개념과 함께 깊게 살펴보며, model의 parameter를 estimate하는 알고리즘인 IRLS(Fisher scoring), HC standard errors의 clustered data 버전인 Cluster-robust standard error를 다룹니다. 3장에서는 GLM에서 여전히 남아있는 observations의 independent 조건(오차항의 독립성)을 극복하여 clustered(panel, longitudinal..) data를 고려할 수 있는 모델인 GEE, GLMM의 근본적인 원리를 M-estimator, robust (sandwich) estimator와 같은 수학적 개념들과 함께 탐구할 것입니다. <strong>{4장은 계획 중에 있습니다.}</strong></p>
<p>결국 “Exploring Regression Models for Regression Analysis”는 Regression Analysis을 논리적으로 수행하기 위해 어떤 Regression Model 들이 있으며, 이들의 수학적인 원리가 무엇인지를 deep dive 하는 글입니다. 영문 글들 중에서도 이 내용들을 이어지게 다뤄주는 글은 거의 없고, 아주 친절하게 수식을 전개하였기 때문에 근본적인 Regression Models의 수학적 원리를 공부하기 좋은 글들이라고 생각하며, R 코드는 간단하게만 주석 형태로 제공하기 때문에 로컬에서 주석 해제 후 돌려보시고, 실제 의학 연구에 관련 분석이 필요하시다면 해당 연구를 <a href="https://www.zarathu.com/en">차라투</a>로 문의해주시길 바랍니다.<br></p>
</section><section id="regression-analysis" class="level2"><h2 class="anchored" data-anchor-id="regression-analysis">1. Regression Analysis</h2>
<p><strong>Regression Analysis</strong>는 통계학에서 가장 널리 사용되는 방법론 중 하나로, 어떤 종속변수(dependent variable)와 하나 이상의 독립변수(independent variables) 간의 관계를 추정하고 해석하는 분석 기법입니다. (종속변수 또한 벡터, 즉 여러 개일 수 있습니다.)</p>
<p>의학 분야에서는 예후를 예측하는 모델을 만들거나(예: 특정 약물 투여 후 혈압, outcome 등의 변화를 예측), 특정 위험인자(risk factor)가 결과에 유의미한 영향을 미치는지(예: 어떤 치료법이 환자의 생존율에 유의미한 차이를 주는지) 등을 살펴보기 위해 <strong>Regression Analysis</strong>가 필수적으로 활용됩니다. 이러한 Regression Analysis을 완벽하게 수행하기 위해서는 본인의 <em>data가 어떤 특성을 갖고 있는지 파악하고</em>, 이에 대해 <em>어떠한 Regression Model을 고려해야 하는지</em> 알고, 이 모델이 <em>어떻게 data에 적합(fit)하는지</em>에 대한 대략적인 수학 또는 알고리즘의 원리와, 적합된 모델이 <em>통계적으로 유의미한지 검정(test)</em>하고 예측 성능이나 해석력을 평가하는 방법까지 전 과정을 이해해야 합니다. 본 1장에서는 가장 간단한, 대신 많은 가정이 필요한 Regression Model인 (General) Linear Regression에 대해서 위 과정과 함께 살펴볼 것이며, 이후의 장들에서는 data의 가정이 조금씩 깨질 때 (data의 특성이 변할 때) 어떤 Regression Model을 고려해야 하는지와 이 Model들의 parameters는 어떻게 수학적으로 추정하는지 살펴볼 것입니다.<br></p>
</section><section id="linear-regression-lm" class="level2"><h2 class="anchored" data-anchor-id="linear-regression-lm">2. Linear Regression (LM)</h2>
<section id="simple-multiple-linear-regression-정의" class="level3"><h3 class="anchored" data-anchor-id="simple-multiple-linear-regression-정의">2.1. (Simple, Multiple) Linear Regression 정의</h3>
<hr>
<p><strong>(General) Linear Regression</strong>은 의학뿐만 아니라 다양한 분야에서 가장 basic한 Regression Model입니다. <strong>Linear Regression</strong>은 독립변수 <img src="https://latex.codecogs.com/png.latex?X"> 와 종속변수 <img src="https://latex.codecogs.com/png.latex?Y"> 간의 <strong>선형(linear) 관계</strong>를 가정하고, 이를 통해 종속변수를 설명하거나 예측합니다. 독립변수가 1개일 때는 <strong>Simple Linear Regression</strong>, 2개 이상일 때는 <strong>Multiple Linear Regression</strong>라고 부릅니다. 의학 연구에서 예를 들면, 혈압(종속변수)을 나이, 체중, 성별(독립변수) 등으로 설명하거나 예측하는 과정을 생각할 수 있습니다. 이러한 Linear regression은 아래 네 가지 가정을 기반으로 추정하며, 이 가정들에 대해서 잘 이해하는 것은 매우 중요하고, 이들 중 특정 가정들이 위배될 경우 이후에 다룰 Regression Model들을 고려해야 함을 명심해주시면 좋을 것 같습니다.</p>
<ul>
<li>
<p><strong>선형성(Linearity) 가정</strong></p>
<p>독립변수와 종속변수가 선형 관계에 있다고 가정합니다. 산점도(scatter plot)를 통해 대략적인 선형 관계 여부를 확인할 수 있습니다.</p>
</li>
<li>
<p><strong>오차항의 정규성(Normality) 가정</strong></p>
<p>주어진 독립변수의 값에서 종속변수의 확률분포는 정규분포를 따른다고 가정합니다. 이는 오차항 ε이 정규분포를 따른다고도 표현할 수 있습니다.(종속변수가 정규분포를 따른다는 뜻은 Linear Model이 종속변수의 mean을 예측하므로, 이 둘의 차인 오차항은 평균이 0이고 분산은 종속변수와 같은 정규분포를 따르게 되기 때문입니다.) 이는 정규 P-P plot 혹은 Q-Q plot 등을 통해 가정 위배 여부를 대략적으로 확인할 수 있습니다.</p>
</li>
<li>
<p><strong>오차항의 독립성(Independence) 가정</strong></p>
<p>각 관측치(Observations, Data(set)) 또는 잔차(residual)가 서로 독립이라고 가정합니다. 즉, data간의 상관관계가 없다고 가정하는 것이고, 잔차산점도(residual plot), Durbin-Watson 통계량 등을 통해 자기상관(autocorrelation)이 있는지 살펴볼 수 있습니다.</p>
</li>
<li>
<p><strong>오차항의 등분산성(Homoscedasticity) 가정</strong></p>
<p>모든 독립변수의 값에서 종속변수의 분산이 동일하다고 가정합니다. 잔차산점도 등을 통해 잔차가 일정한 분산을 가지는지 대략적으로 확인할 수 있습니다.</p>
</li>
</ul>
<p>이후 Linear Regression Model은 다음과 같은 과정으로 분석을 수행하게 됩니다; (1) 최소제곱법(Least Squares)를 통해 model parameters를 추정(estimate), (2) 결정계수(<img src="https://latex.codecogs.com/png.latex?R%5E2">)를 통해 모델이 종속변수를 얼마나 잘 설명하는지 확인, (3) F-검정(F-test)으로 전체 회귀식의 유의성을 검정, (4) 검정(t-test)으로 각 회귀계수(regression coefficient)가 유의미한지 확인. 특히, 독립변수 각각의 Model coefficient(parameter)를 검정 함으로써 종속변수와의 상관성을 분석하는 (4) 과정은 유의성을 판단하는 가장 중요한 검정 과정으로, 고전적으로 <strong>Wald Test, Likelihood Ratio Test, Score Test</strong>가 자주 사용됩니다.</p>
</section><section id="linear-regression-수학적-표현-및-추정" class="level3"><h3 class="anchored" data-anchor-id="linear-regression-수학적-표현-및-추정">2.2. Linear Regression 수학적 표현 및 추정</h3>
<hr>
<p>Linear Regression은 다음과 같은 형태를 가집니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY_i%20=%20%5Cbeta_0%20+%20%5Cbeta_1%20X_%7Bi1%7D%20+%20%5Cbeta_2%20X_%7Bi2%7D%20+%20%5Ccdots%20+%20%5Cbeta_k%20X_%7Bik%7D%20+%20%5Cvarepsilon_i%0A"></p>
<p>혹은 Matrix 형태로 간단히 쓰면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D%20=%20%5Cmathbf%7BX%7D%20%5Cboldsymbol%7B%5Cbeta%7D%20+%20%5Cboldsymbol%7B%5Cvarepsilon%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0Awhere,%20%5Cquad%20%5Cmathbf%7By%7D%20%5Cin%20%5Cmathbb%7BR%7D%5En,%20%5Cmathbf%7BX%7D%0A%5Cin%20%5Cmathbb%7BR%7D%5E%7Bn%20%5Ctimes%20p%7D,%20%5Cboldsymbol%7B%5Cbeta%7D%20%5Cin%20%5Cmathbb%7BR%7D%5Ep,%0A%5Cboldsymbol%7B%5Cvarepsilon%7D%20%5Cin%20%5Cmathbb%7BR%7D%5En%0A"></p>
<ul>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7By%7D%20%5Cin%20%5Cmathbb%7BR%7D%5En">: 관측된 종속변수들의 벡터</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D%20%5Cin%20%5Cmathbb%7BR%7D%5E%7Bn%20%5Ctimes%20p%7D">: 설계 행렬, 각 행은 하나의 관측치(Observation)이며 (<img src="https://latex.codecogs.com/png.latex?n">), 각 열은 하나의 독립변수에 (<img src="https://latex.codecogs.com/png.latex?p">) 해당</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D%20%5Cin%20%5Cmathbb%7BR%7D%5Ep">: 추정하고자 하는 회귀 계수(or 모수 or parameters) 벡터 (여기선 intercept <img src="https://latex.codecogs.com/png.latex?%5Cbeta_0"> 제외)</p></li>
<li><p><img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D%20%5Cin%20%5Cmathbb%7BR%7D%5En">: 랜덤 오차 항 벡터, 위 <strong>네 가정</strong>에 따라 랜덤 오차항은 평균이 <img src="https://latex.codecogs.com/png.latex?E(%5Cboldsymbol%7B%5Cvarepsilon%7D)%20=%200">이고, 분산이 모든 관측치에서 같으며 독립인 정규분포</p></li>
</ul>
<p>입니다. Multiple Linear Regression의 모델 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">의 분산은 scalar variance가 아니라 (co)variance matrix 형태가 될 것입니다. (simple도 intercept를 고려하면 matrix) 앞으로 세 장에 걸쳐 이를 분산 행렬이라고 부르겠지만, 정확히는 <strong>분산-공분산 행렬</strong>이라고도 부릅니다. 또한, 위 matrix form 수식처럼 intercept를 제외하는 경우도 많은데, 항상 있다고 가정하며 가독성을 위해 제외한 것이니 혼동할 필요 없이 자연스럽게 읽으시면 됩니다. Covariance matrix에서는 회귀계수 서로 간의 공분산이 들어있으므로, 각 <strong>회귀계수(독립변수의 유의성)에 대한 검정</strong>은 <strong>Covariance Matrix에 있는 각 diagonal 원소들을 사용하여 수행</strong>하게 됩니다. (<strong>회귀계수 각각에서 스스로의 대한 공분산 = 회귀계수의 분산</strong>) 예를 들어 <img src="https://latex.codecogs.com/png.latex?%5Cbeta_2">의 분산은 <img src="https://latex.codecogs.com/png.latex?p">가 5 (독립변수가 4개, intercept 포함)일 때 Covariance Matrix의 3행 3열 값이 될 것입니다. (intercept가 없다면 2행 2열) 여기서 확인할 수 있듯 Regression Model의 <strong>parameter 추정 역시 중요</strong>하지만, <strong>parameter의 Covariance Matrix를 추정하는 것 또한 유의성 판단에서 매우 중요</strong>하며, 이후의 장들 또한 자연스럽게 Model의 소개, paramater 추정법, parameter의 (Co)variance Matrix를 추정법으로 이루어지게 될 것입니다.</p>
<p>또한, Linear Model 식은 선형대수학에서 <strong>Hyper Plane</strong>으로 정의하는 형태가 되는데, 쉽게 설명하자면 독립변수가 하나일 경우 종속변수와의 2차원 평면에서 직선(2차원의 Hyper Plane)을 띄고, 독립변수가 두 개일 경우 종속변수와의 3차원 공간에서 평면을 띄는(3차원의 Hyper Plane), 선형적으로 공간을 두 부분으로 가르는 공간이라고 생각하시면 될 것 같습니다. (마찬가지로 쉽게 생각하면, 비선형적일 경우 직선이 아니라 곡선, 평면이 아니라 곡면일 것입니다.)</p>
<p>이제 어떻게 Linear Model을 regression, 즉 fit(parameter를 추정)할 것인지 설명하겠습니다. 가장 기본적으로 알려진 Model의 통계학적 parameter 추정법으로는 <strong>Ordinary Least Squares(OLS), Maximum Likelihood Estimation(MLE), Method of Moment(MOM)</strong>가 있습니다. 이후의 모델들은 대부분 MLE로 모델을 추정하지만, Linear Model은 특별하게도 MLE와 OLS의 추정 결과가 같고, OLS의 추정 결과는 BLUE(Best Linear Unbiased Estimator) 입니다. <strong>최소제곱법(OLS, Ordinary Least Squares) 추정량(estimator)</strong>은 <strong>오차 제곱합(SSR, Sum of Squared Residuals)</strong>, 즉 모든 데이터에서 실제 종속변수 값과 모델의 예측 값의 차이를 제곱한 수들의 합을 최소화하는 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">를 찾는 것이며, 다음의 closed-form solution을 갖습니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%20=%20%5Cleft(%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D%20%5Cright)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7By%7D%0A"></p>
<p>그리고, 고전적 가정(특히 등분산성, 독립성, 정상성)이 모두 충족된다고 할 때, 이 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">의 추정오차의 공분산행렬(covariance matrix)은 <img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20%5Csigma%5E2%20%5Cleft(%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D%20%5Cright)%5E%7B-1%7D%0A"> 이며, 실제 계산할 때는 데이터에서 추정한 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Csigma%7D%5E2">를 통해 <img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Coperatorname%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20%5Chat%7B%5Csigma%7D%5E2%20%5Cleft(%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D%20%5Cright)%5E%7B-1%7D,%20%5Cquad%20where%20%5C;%20%5Chat%7B%5Csigma%7D%5E2%20=%20%5Cfrac%7B%5Csum%20e_i%5E2%7D%7BN-K%7D,%20%5C;%20e_i%20=%20y_i%20-%20x_i%20%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%0A"></p>
<p>로 추정합니다. (분모에서 K를 뺀 이유는 degree of freedom에 의한 것이며, 에러 term이 정확히는 차원에 맞춰 <img src="https://latex.codecogs.com/png.latex?x_i%5E%5Ctop%20%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D"> 지만, 앞으로 이정도 표기는 가독성을 위해 넘어가겠습니다.) 이렇게 구한 모델 추정량과 추정오차의 공분산행렬을 통해 각 회귀계수에 대한 검정을 수행하거나 신뢰구간(CI)을 계산할 수 있게 됩니다.</p>
<p>다음으로 넘어가기 전에, Linear Regression에서 (1) <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D">이 closed-form solution <img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7By%7D">을 갖게 되는 과정과, (2) 위에서 언급하였듯 model의 parameter (<img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D">) OLS 추정량이 MLE(Maximum likelihood estimation) 추정량과 같음을 증명하겠습니다.</p>
<p><strong>(1) Prove</strong> <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7By%7D"><strong>.</strong></p>
<p>우선, OLS 추정량은 오차 제곱합(SSR)을 최소화하는 식이므로 오차 SSR을 표현하는 식 <img src="https://latex.codecogs.com/png.latex?J(%5Cbeta)">를 적으면 다음과 같습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AJ(%5Cbeta)%20=%20%5Cfrac%7B1%7D%7B2%7D%20(X%5Cbeta%20-%20y)%5E%5Ctop%20(X%5Cbeta%20-%20y)%20=%20%5Cfrac%7B1%7D%7B2%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cbig(%20x_i%20%5Cbeta%20-%20y%5E%7B(i)%7D%20%5Cbig)%5E2%0A"></p>
<p>이제 위 <img src="https://latex.codecogs.com/png.latex?J(%5Cbeta)">를 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">에 대해 미분하였을 때 0이 나오는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D">이 SSR이 최소가 되는 OLS 추정량입니다.(이에 대한 증명은 안하겠지만, 간단하게 볼록한 2차 함수에서는 미분값이 0인 점이 최소점인 것과 같은 원리라고 생각하시면 되겠습니다.) 식을 풀어보면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cnabla_%5Cbeta%20J(%5Cbeta)%20=%20%5Cnabla_%5Cbeta%20%5Cfrac%7B1%7D%7B2%7D%20(X%5Cbeta%20-%20y)%5E%5Ctop%20(X%5Cbeta%20-%20y)%20%5C%5C%0A=%20%5Cfrac%7B1%7D%7B2%7D%20%5Cnabla_%5Cbeta%20%5Cbig(%20(X%5Cbeta)%5E%5Ctop%20X%5Cbeta%20-%20(X%5Cbeta)%5E%5Ctop%20y%20-%20y%5E%5Ctop%20(X%5Cbeta)%20+%20y%5E%5Ctop%20y%20%5Cbig)%20"> 입니다. <img src="https://latex.codecogs.com/png.latex?%5Cnabla">는 미분하는 변수가 scalar가 아니라 vector이기 때문에 사용되는 기호이며(gradient), 그냥 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">로 미분한다고만 생각하시면 됩니다. 이후 미분하는 과정 또한 크게 다르지 않습니다. 식을 계속 진행해보면, <img src="https://latex.codecogs.com/png.latex?y%5E%5Ctop%20y">는 식에 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">가 없고 <img src="https://latex.codecogs.com/png.latex?(X%5Cbeta)%5E%5Ctop%20y">와 <img src="https://latex.codecogs.com/png.latex?y%5E%5Ctop%20(X%5Cbeta)"> 둘다 단순히 두 벡터의 내적인 똑같은 식이므로 <img src="https://latex.codecogs.com/png.latex?%20=%20%5Cfrac%7B1%7D%7B2%7D%20%5Cnabla_%5Cbeta%20%5Cbig(%20%5Cbeta%5E%5Ctop%20(X%5E%5Ctop%20X)%20%5Cbeta%20-%202%20(X%5E%5Ctop%20y)%5E%5Ctop%20%5Cbeta%20%5Cbig)%20">이고, 이제 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">로 미분하면, 우항은 그냥 미분, 좌항은 두 번 나오므로 곱미분을 수행하여<br><img src="https://latex.codecogs.com/png.latex?%20=%20%5Cfrac%7B1%7D%7B2%7D%20%5Cbig(%202%20X%5E%5Ctop%20X%20%5Cbeta%20-%202%20X%5E%5Ctop%20y%20%5Cbig)%20"><br><img src="https://latex.codecogs.com/png.latex?%20=%20X%5E%5Ctop%20X%20%5Cbeta%20-%20X%5E%5Ctop%20y%20=%200%20"></p>
<p>입니다. 결국 OLS 추정량 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D">는</p>
<p><img src="https://latex.codecogs.com/png.latex?%20X%5E%5Ctop%20X%20%5Chat%7B%5Cbeta%7D%20=%20X%5E%5Ctop%20y%20"> 이고, 양변 앞에 역행렬을 곱해주면<br><img src="https://latex.codecogs.com/png.latex?%20%5Chat%7B%5Cbeta%7D%20=%20(X%5E%5Ctop%20X)%5E%7B-1%7D%20X%5E%5Ctop%20y%20">가 됩니다.</p>
<p><strong>(2) Prove OLS estimator is same as MLE estimator in Linear Model (Regression).</strong></p>
<p><br>
위 Linear Regression Model 식을 다음과 같이 작성할 수 있습니다. <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D%20=%20%5Cmathbf%7BX%7D%20%5Cboldsymbol%7B%5Cbeta%7D%20+%20%5Cboldsymbol%7B%5Cvarepsilon%7D,%20%5Cquad%20%5Cboldsymbol%7B%5Cvarepsilon%7D%20%5Csim%20%5Cmathcal%7BN%7D(%5Cmathbf%7B0%7D,%20%5Csigma%5E2%20%5Cmathbf%7BI%7D)%0A"></p>
<p>식이 어색할 수 있지만 정확히 이전의 수학적 표현과 동일하며, <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D"> term이 복잡해 보이는 이유는, 이전의 네 가지 조건들을 수식으로 반영하여 모델을 확률 기반으로 해석하였기 때문입니다. 식을 설명드리자면, 선형성(Linearity) 가정에 의해 error의 mean이 (기댓값이) 0이 되고, 오차항의 정규성 가정에 의해 정규 분포를 따르며, 오차항의 독립성 및 등분산성 가정에 의해 분산(Covariance Matrix)이 값이 모두 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">인 대각행렬, 즉 다른 observations간의 correlation은 0이고(없고), 각 observations의 분산은 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">로 일정함을 표현한 것입니다.</p>
<p><strong>최대가능도방법</strong> 또는 <strong>최대우도법</strong>(Maximum Likelihood Method)은 어떤 모수와 표집한 값들이 주어졌을 때, 표집한 값들이 나올 가능도(확률)을 최대로 만드는 모수를 선택하는 아주 general한 모델의 점 추정 방법이며, 처음 들어보셨다면 대표님께서 더욱 자세하게 설명하신 블로그 글을 읽으시면 좋을 것 같습니다. 철학적으로는 종속변수의 확률 분포를 데이터에 맞게 가정한 후, 모든 observations(data) 각각이 나올 확률(분포)을 모두 곱한 식이 최대가 되도록 하는 모수값이 바로 MLE를 통해 추정한 parameter라고 생각하시면 됩니다. 간단한 예시를 들어보자면, 동전을 던져 앞면이 7번, 뒷면이 3번 나온 데이터에서 동전이 앞면이 나올 확률을 모수로 두고 종속변수의 분포를 베르누이 분포로 가정하면, 각각의 동전을 던져 얻은 관측치들 10개의 data의 Likelihood(확률)를 모두 곱하면 모수가 하나였으니 변수가 하나인 하나의 식(함수)가 나오고, 이 식(함수)을 최대화 하는 모수를 찾는 method가 MLE라고 간단하게 정리할 수 있을 것 같습니다. (이때 추정된 결과는 0.7일 것이고, 모수로 식을 미분한 함수(score function)이 0인 값을 찾음으로써 모수를 추정하게 되며, MLE 이외에도 MAP, 베이지안 등 다른 추정 기법들도 있지만 여기에서는 MLE 정도를 이해하면 충분할 것 같습니다.)</p>
<p>이제 돌아와서 LM에서는 OLS estimator와 MLE estimator가 상응함을 보이겠습니다. 이후의 식 전개는 dim이 1일 때로 증명하겠습니다. Multi-dimension에서는 분포식과 term이 더 복잡하지만 meaning은 1차원과 정확히 동일하기 때문입니다. 오차 항 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D">는 정규분포를 따르므로, n개의 관측치에 대한 Likelihood는 <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathcal%7BL%7D(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Cprod_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%20%5Cpi%20%5Csigma%5E2%7D%7D%20%5Cexp%5Cleft(%20-%5Cfrac%7B(y_i%20-%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%7D%7B2%20%5Csigma%5E2%7D%20%5Cright)%0A"></p>
<p>이며, 로그를 씌운 log Likelihood는 <img src="https://latex.codecogs.com/png.latex?%0A%5Cell(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Clog%20%5Cmathcal%7BL%7D(%5Cboldsymbol%7B%5Cbeta%7D)%0A"></p>
<p>입니다. 식이 복잡해 보이지만 데이터 n개에 대하여 정규(가우시안)분포를 따르는 종속변수가 모델의 예측값을 가질 확률을 단순히 모두 곱한 식입니다. 결국 <strong>얻어진 표본 data 자체가 나올 확률에 대한 식을 세운 후, 이를 최대화 하는 parameter를 찾는 과정이 MLE estimation</strong>이라고 직관적으로 해석할 수 있습니다.</p>
<p>돌아와서, log를 씌운 이유를 생각해보겠습니다. log의 그래프를 생각해보시면 log함수는 직관적으로도 monotonically increase하기 때문에 likelihood를 최대화 하는 parameter와 log likelihood를 최대화 하는 parameter는 같으며, log는 곱셈을 덧셈으로, 지수 term을 아래로 만들어주어 likelihood 식이 매우 간단해지기 때문에 (아래에서 실제로 그런지 확인해보세요.) MLE에서는 항상 log likelihood를 최대화하는 parameter를 찾는 방향으로 parameter를 추정합니다. 이어서 전개하면,<br><br><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Clog%20%5Cprod_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%20%5Cpi%20%5Csigma%5E2%7D%7D%20%5Cexp%5Cleft(%20-%5Cfrac%7B(y_i%20-%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%7D%7B2%20%5Csigma%5E2%7D%20%5Cright)%0A"><br><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Clog%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%20%5Cpi%20%5Csigma%5E2%7D%7D%20-%20%5Cfrac%7B(y_i%20-%20%5Cmathbf%7Bx_i%5E%5Ctop%7D%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%7D%7B2%20%5Csigma%5E2%7D%0A"><br><img src="https://latex.codecogs.com/png.latex?%0A=%20n%20%5Clog%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%20%5Cpi%20%5Csigma%5E2%7D%7D%20-%20%5Cfrac%7B1%7D%7B2%20%5Csigma%5E2%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20(y_i%20-%20%5Cmathbf%7Bx_i%5E%5Ctop%7D%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%0A"></p>
<p>입니다. Log likelihood를 최대화 하는 것은 negative Log likelihood를 최소화 하는 것과 같으므로 양변에 -를 곱하면<br><img src="https://latex.codecogs.com/png.latex?%0A-%20%5Cell(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20-n%20%5Clog%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%20%5Cpi%20%5Csigma%5E2%7D%7D%20+%20%5Cfrac%7B1%7D%7B2%20%5Csigma%5E2%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20(y%5E%7B(i)%7D%20-%20%5Cmathbf%7Bx%7D%5E%7B(i)%5E%5Ctop%7D%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%0A"></p>
<p>이고, 상수 <img src="https://latex.codecogs.com/png.latex?-n%20%5Clog%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%20%5Cpi%20%5Csigma%5E2%7D%7D">는 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">에 영향을 주지 않으므로 제거하면, <img src="https://latex.codecogs.com/png.latex?%0A-%20%5Clog%20%5Cmathcal%7BL%7D(%5Cboldsymbol%7B%5Cbeta%7D)%20%5Capprox%20%5Cfrac%7B1%7D%7B2%20%5Csigma%5E2%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20(y%5E%7B(i)%7D%20-%20%5Cmathbf%7Bx%7D%5E%7B(i)%5E%5Ctop%7D%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%0A"></p>
<p>가 됩니다. 아까 보았듯 OLS estimator는 SSR을 최소화하는 아래 문제로 정의되므로, <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D_%7B%5Ctext%7BOLS%7D%7D%20=%20%5Carg%20%5Cmin_%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20(y%5E%7B(i)%7D%20-%20%5Cmathbf%7Bx%7D%5E%7B(i)%5E%5Ctop%7D%20%5Cboldsymbol%7B%5Cbeta%7D)%5E2%0A"></p>
<p>결국 위 두 식, OLS estimator와 MLE estimator의 목적식이 같음을 알 수 있고(종속변수와 모델 예측의 차이의 제곱을 모든 data point에서 더한 값을 최소화 하는 beta), 따라서 추정량 또한 같습니다. <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D_%7B%5Ctext%7BMLE%7D%7D%20=%20%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D_%7B%5Ctext%7BOLS%7D%7D%0A"></p>
<p>위 증명은 MLE로 LM의 parameter를 추정한 것이 아니라 식 전개를 통해 OLS와 목적식이 같음을 보인 것이고, 실제 MLE로 parameter를 추정하는 과정은 2장부터 자세히 보게 될 것입니다. 한 가지 첨언하자면, Linear model에서는 이 두 추정량이 같지만, 정규성 가정이 깨진 경우 두 추정량은 같지 않을 수 있습니다. 이유를 생각해보면, 정규분포는 확률이 가장 큰, 즉 mode가 mean이며 mean을 기준으로 양 옆이 symmetric한 분포이기 때문에 두 추정량이 같으며, 정규분포가 아닐 경우 mean을 추정하려고 하는 MLE는 절대적인 거리를 좁히려는 OLS와 다른 값을 추정하게 된다고 직관적으로 생각해볼 수 있습니다.<br></p>
</section></section><section id="linear-regression-in-homo-vs.-heteroskedasticity" class="level2"><h2 class="anchored" data-anchor-id="linear-regression-in-homo-vs.-heteroskedasticity">3. Linear Regression in Homo vs.&nbsp;Heteroskedasticity</h2>
<p>앞서 언급한 Linear Regression 모형은 “오차항의 등분산성(homoscedasticity)”을 가정합니다. 그러나 실제 연구 상황에서는 이 가정이 위배되는 경우가 흔합니다. <strong>Heteroskedasticity</strong>는 이러한 오차항은 독립 변수에 따라 같지 않을 수 있다는 뜻으로, 등분산성 가정이 깨진 경우를 의미합니다. 즉,</p>
<ul>
<li>
<strong>Homoscedasticity</strong>: 모든 관측값에 대해 오차항(또는 종속변수)의 분산이 동일</li>
<li>
<strong>Heteroskedasticity</strong>: 오차항(또는 종속변수)의 분산이 관측값에 따라 다름</li>
</ul>
<p>입니다. 즉 이전 수식 중 오차 항을 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D">로 두었다면, observations의 분산은 더이상 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">로 일정하지 않고 observations마다 다른 값을 갖을 수 있다는 뜻입니다. 예를 들어, 임상 연구에서 나이(age)에 따라 혈압(blood pressure)의 분산이 달라질 수 있습니다. 이런 상황에선 등분산성을 가정하는 것이 부적절해집니다. 등분산성 가정이 깨진 상황에서 Linear regression을 수행하면, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D"> 자체가 편향되진 않더라도(즉, 일관성(consistency)은 여전히 유지), <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D"> 의 분산 추정치 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Csigma%7D%5E2%20(%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D%20)%5E%7B-1%7D"> 가 bias를 갖게 되어 잘못된 표준오차, 잘못된 유의성 검정 결과로 이어질 수 있습니다. 따라서, <strong>Heteroskedasticity가 의심되는 상황에서</strong>는 모델의 분산을 <strong>안정적으로 추정</strong>해야 하며, 이때 1번으로 고려되는 방법 중 하나가 <strong>Heteroskedasticity-consistent(heteroskedasticity-robust) standard errors</strong>, 줄여서 HC standard errors를 통한 (Co)variance Matrix estimation입니다.<br></p>
</section><section id="heteroskedasticity-consistent-standard-errors-hc-standard-errors" class="level2"><h2 class="anchored" data-anchor-id="heteroskedasticity-consistent-standard-errors-hc-standard-errors">4. Heteroskedasticity-Consistent Standard Errors (HC Standard Errors)</h2>
<section id="hc-standard-errors-정의" class="level3"><h3 class="anchored" data-anchor-id="hc-standard-errors-정의">4.1. HC Standard Errors 정의</h3>
<hr>
<p><strong>HC(Heteroskedasticity-Consistent) standard errors</strong>는, 고전적 선형회귀 가정 중 <strong>등분산성</strong>만 깨졌을 때(나머지 <em>선형성, 정규성, 독립성 가정 in LM</em> 은 그대로 유지) <strong>회귀계수 추정치의 분산</strong>을 “강건(robust)”하게 추정하기 위한 method입니다. 이는 heteroskedasticity-robust standard errors 또는 HCCME(Heteroskedasticity-Consistent Covariance Matrix Estimation) 라고도 부르며, 이전에로 구한 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D">를 그대로 사용하되, 그 공<strong>분산행렬 추정만 새롭게(robust하게) 구하는 방식</strong>입니다. 다만, 독립성(오차항들이 서로 독립), 선형성, 정규성 등의 다른 가정이 또 깨져 있다면 HC SE만으로는 대응할 수 없음을 명심하시길 바랍니다. 예를 들어, 오차항이 서로 상관되어 있는 clustered data에서는 더 이상 추정량이 consist하지 않기 때문에, 이보다 한 단계 더 나아간 cluster-robust standard errors, 혹은 GEE, GLMM 등의 모델을 고려해야 합니다. 즉, “등분산성 가정만 깨졌을 때” 쓰는 표준오차(or covariance matrix) 추정량이라고 생각하면 됩니다. 또한, 모델이 실제로 크게 잘못 설정되어 있다면(<em>선형성조차 안 맞는 경우, 모델의 estimator가 크게 편향, HC se가 모델에서 구한 se와 크게 차이가 나는 경우 등</em>), 그때는 variance(standard error)만 “robust standard errors”로 바꾼다고 해서 문제가 해결되지 않고, 모델을 수정하거나 data를 다시 확인하고 분석을 다른 방식으로 수행해야 합니다.</p>
<p>즉 Greene의 말처럼,</p>
<blockquote class="blockquote">
<p>“Simply computing a robust covariance matrix for an otherwise inconsistent estimator does not give it redemption.”</p>
</blockquote>
<p>이라는 점을 늘 유의해야 합니다. 정리하자면, 모델을 잘 구성하고, 모델의 분산과 큰 차이가 없을 경우에는 <em>모델의 parameter의 분산 효율성이 최대가 아니거나 MLE 추정을 통해 얻은 모델의 분산이 더 이상 불편향이 아닐 때도</em> heteroskedasticity-robust standard errors를 통해서 모델의 분산을 robust하게 estimate할 수 있으며, 분석 결과 편향이 너무 크지 않다면 이후의 검정에서 이 HC se를 사용함으로써 이후의 통계 분석에서 설득력을 높일 수 있습니다. 추가로 스포하자면, 아래에서 다룰 Heteroskedasticity-robust Standard Errors 식은 Linear Model에서 robust하게 분산을 추정하는 <strong>Robust(or Sandwich) Estimation</strong>이며, 이후의 모델에서도 해당 모델에 대한 Sandwich Estimator를 고려함으로써 안정적인 분산 추정 방법에 대해 다룰 것입니다.</p>
</section><section id="hccme0-수학적-표현" class="level3"><h3 class="anchored" data-anchor-id="hccme0-수학적-표현">4.2. HC(CME)0 수학적 표현</h3>
<hr>
<p>이제 위에서 설명한 Heteroskedasticity한 data에서 robust하게 standard errors, 혹은 Covariance Matrix를 estimate 하는 heteroskedasticity-consistent standard errors의 수식을 유도하겠습니다.</p>
<p>이전에 선형 회귀 모델을 다음과 같이 정의한 적이 있습니다.<br><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7By%7D%20=%20%5Cmathbf%7BX%7D%20%5Cboldsymbol%7B%5Cbeta%7D%20+%20%5Cboldsymbol%7B%5Cvarepsilon%7D%0A"> 이때, 위에서는 등분산성을 가정하였기 때문에 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D">를 원소값이 모두 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2"> 인 대각행렬로 가정하여 각 observations의 분산은 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">로 일정함을 가정했다면, 이번에 정의한 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D">은 <img src="https://latex.codecogs.com/png.latex?E(%5Cboldsymbol%7B%5Cvarepsilon%7D)%20=%200">, <img src="https://latex.codecogs.com/png.latex?E(%5Cboldsymbol%7B%5Cvarepsilon%7D%5Cboldsymbol%7B%5Cvarepsilon%7D%5E%5Ctop)%20=%20%5CPhi"> 입니다. 이때 <img src="https://latex.codecogs.com/png.latex?%5CPhi">는 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7Bdiag%7D(%5Cvarepsilon_i%5E2)">로 정의되는 대각행렬이며(대각 성분을 제외하면 모두 0인 행렬), 식을 통해서 여전히 error term의 mean이 0이고 observations들은 independent하지만(대각행렬이므로), 더 이상 observations의 분산이 서로 일치하지 않음을 표현한 것입니다. 즉, 이제 Homoscedasticity에서 Heteroskedasticity을 고려하기 시작했다는 것을 수식을 해석함으로써 알 수 있습니다.</p>
<section id="variance-estimatation" class="level4"><h4 class="anchored" data-anchor-id="variance-estimatation">Variance estimatation</h4>
<p>위 식의 OLS 추정량은 여전히 <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7By%7D%0A"></p>
<p>입니다. (still consistent) 이제 모델의 variance 식을 전개해보면, <img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20%5Coperatorname%7BVar%7D%20%5Cleft%20(%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7By%7D%20%5Cright%20)"></p>
<p>이며, <img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop">는 determinant한 값으로, variance는 식 내의 determinant한 salar는 Var 밖으로 제곱과 함께 나오는 것처럼 (<img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7BVar%7D(aX)%20=%20a%5E2%5C,%5Cmathrm%7BVar%7D(X)">), 행렬 또는 벡터에 대해도 비슷하게 <img src="https://latex.codecogs.com/png.latex?%0A=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7BVar%7D%20(%20%5Cmathbf%7By%7D%20)%20%5Cleft%20((%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cright%20)%20%5E%20%5Ctop%20%5C%5C%0A=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7BVar%7D%20(%20%5Cmathbf%7By%7D%20)%20%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D"> 가 됩니다. (벡터와 행렬을 다룰 때에는 위처럼 제곱한다고 생각하시면 될 것 같습니다.) 참고로, <img src="https://latex.codecogs.com/png.latex?%5Cleft%20(%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cright%20)%5E%20%5Ctop%20=%20%5Cleft%20(%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%5Ctop%20%5Cright%20)%5E%20%7B-1%7D%20=%20%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%20%7B-1%7D"> 입니다.</p>
<p>따라서, 처음 설정한대로 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D%20(%20%5Cmathbf%7By%7D%20)%20=%20%5CPhi">를 넣으면, 최종적으로 추정된 consistent model <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">의 (co)variance matrix <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)">는 다음과 같습니다. <img src="https://latex.codecogs.com/png.latex?%0A(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5CPhi%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>하나 확인할 수 있는 사실은, 등분산성 가정(<img src="https://latex.codecogs.com/png.latex?%5CPhi%20=%20%5Csigma%5E2%20%5Cmathbf%7BI%7D">) 하에선 위 식이 아래처럼 단순화되어 이전 (co)variance matrix가 나왔던, 가정하의 특수한 결과라는 것입니다. <img src="https://latex.codecogs.com/png.latex?%20%5Coperatorname%7BVar%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%5E%20%7B-1%7D%20%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5CPhi%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%20%5E%20%7B-1%7D%20%5C%5C%20=%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%20%5E%20%7B-1%7D%20%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Csigma%20%5E%202%20%5Cmathbf%7BI%7D%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%20%5E%20%7B-1%7D%20%5C%5C%20=%20%5Csigma%20%5E%202%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%20%5E%20%7B-1%7D%20%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%20%5E%20%7B-1%7D%20%5C%5C%20=%20%5Csigma%20%5E%202%20(%5Cmathbf%7BX%7D%20%5E%20%5Ctop%20%5Cmathbf%7BX%7D)%20%5E%20%7B-1%7D%20"> 또한, 이 때의 (OLS 추정량)의 분산 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">의 식도 그냥 보여드리고 넘어갔었는데, 이는 대각 성분이 모두 같기 때문에 단순히 <strong>잔차 제곱합</strong> <img src="https://latex.codecogs.com/png.latex?%5Csum%20e_i%5E2">을 자유도(<img src="https://latex.codecogs.com/png.latex?N-K">)로 나눈 값(<img src="https://latex.codecogs.com/png.latex?s%5E2">)으로 추정하여 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Csigma%7D%5E2%20=%20%5Cfrac%7B%5Csum%20e_i%5E2%7D%7BN-K%7D"> 로 한번에 표현한 식이었다는 것도 확인할 수 있습니다.</p>
</section><section id="hc0s-robustness" class="level4"><h4 class="anchored" data-anchor-id="hc0s-robustness">HC0’s Robustness</h4>
<p>이분산성이 존재할 때 <img src="https://latex.codecogs.com/png.latex?%5CPhi">는 대각행렬이지만 <strong>대각원소가 서로 다릅니다</strong>. White는 1980년에 이 heteroskedasticity에 robust하게 분산을 추정하기 위해 위 식에서 <img src="https://latex.codecogs.com/png.latex?%5CPhi%20=%20%5Csigma%5E2%20%5Cmathbf%7BI%7D">로 term이 소거되게 하는 대신, <img src="https://latex.codecogs.com/png.latex?%5CPhi_%7Bii%7D">를 <strong>잔차 제곱</strong> <img src="https://latex.codecogs.com/png.latex?e_i%5E2">로 추정하는 HC0을 제안했습니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5CPhi%7D_%7Bii%7D%20=%20e_i%5E2%20%5Cquad%20%5CRightarrow%20%5Cquad%20%5Chat%7B%5CPhi%7D%20=%20%5Coperatorname%7Bdiag%7D(e_i%5E2)%0A"></p>
<p>이를 위 (co)variance 식에 대입하면 HC0 분산 추정량이 얻어집니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BHC0%7D%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7Bdiag%7D(e_i%5E2)%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>이는 variance 식으로부터 시작해서 얻은 식이고, 데이터로부터 얻은 matrix <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5CPhi%7D%20=%20%5Coperatorname%7Bdiag%7D(e_i%5E2)">를 사용하여 이분산성의 구체적 형태를 가정하지 않았기 때문에 안정적으로 heteroskedasticity를 고려합니다. 이처럼 <strong>앞 뒤가 같은 모양 사이에 데이터로부터 얻은 분산 추정 식이 들어간 형태가 샌드위치 같다고 해서 Sandwich Estimator</strong>라고도 부르며, 여전히 <strong>가운데 matrix 항이 diagonal 함</strong>을 통해 오차(또는 관측치)간의 <strong>상관관계는 없다</strong>는 것도 기억해야 합니다.</p>
</section></section><section id="hccme-13-수학적-표현" class="level3"><h3 class="anchored" data-anchor-id="hccme-13-수학적-표현">4.3. HC(CME) 1~3 수학적 표현</h3>
<hr>
<p>이 HC Standard Error가 나온 이후, 위 식으로부터 구한 분산은 소표본에서는 <strong>모델의 overfitting(과적합, 딥러닝과 머신러닝에서도 자주 대두되는 문제로 데이터가 적어 학습한 데이터에서는 오차가 지만 보지 못한 데이터에 대해서는 오차가 크게 나오는 상황, 통계 분석은 test data를 따로 두지 않기 때문에 overfitting 대처가 더욱 중요.)</strong>에 의해 residual이 작게 나오고, 이에 따라 <strong>모델의 covariance가 과소평가</strong>될 가능성이 크다는 문제가 제시되었습니다. 이에, 소표본에서도 안정적으로 추정하기 위해 White(1980)의 HC Standard Errors를 HC0로 두고, 여러 버전의 HC Standard Errors가 최근까지 다양한 학자들에 의해 연구되고 있습니다. 보통 HC0~HC5까지를 HC Standard Errors로 부르며, HC 오른쪽 숫자가 버전을 뜻하여 이 수가 커질수록 최근에 나온 것이고, 각각은 simulation 실험을 통해 소표본에서 더욱 강건함을 보이는 식으로 논문이 나옵니다. 중요한 점은, 첫 번째로 이 <strong>HC Standard Errors들은 소표본에서는 값이 다르지만 표본의 크기가 무수히 커질수록 asymptotically하게 같으며</strong> 소표본에서의 고려를 위해 값을 더 크게 만드는 추가 term을 넣어준다는 점(물론 수식적으로 필요성을 증명하지만), 둘 째로 R의 sandwich 패키지에서는 4까지 지원하고, 3을 넘는 HC 시리즈가 쓰이는 경우는 거의 볼 수 없기 때문에 3까지의 식 정도만 다루어도 충분하다는 점, 마지막으로 위 직관적인 이유로나 수식적으로나 식을 해석해 보면 거의 모든 표본 cases에 대해 <strong>버전이 클수록 분산을 더욱 크게 추정한다는 점</strong>입니다. <del>따라서 유의성을 보이기 위해서는 버전이 작은게 유리할 것입니다.^^</del> 아래에서는 HC1부터 3까지의 수식을 살펴보겠습니다. 각각의 철학에 대한 자세한 증명 과정 또한 다룰까 했지만 중요하지 않기 때문에 간단히 소개한 후 로컬에서 돌려 보실 R 예시 코드를 보여드릴 것입니다.</p>
<section id="hc1-소표본-편향-보정" class="level4"><h4 class="anchored" data-anchor-id="hc1-소표본-편향-보정">HC1: 소표본 편향 보정</h4>
<p>HC1은 소표본에서 자유도 조정 인자 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Bn%7D%7Bn-k%7D">를 도입하여 편향을 줄인다는 철학에서 출발하여, 잔차 <img src="https://latex.codecogs.com/png.latex?e_i%5E2">의 기대값이 <img src="https://latex.codecogs.com/png.latex?%5Csigma_i%5E2(1-h_%7Bii%7D)">이므로, <img src="https://latex.codecogs.com/png.latex?E(e_i%5E2)%20%5Capprox%20%5Csigma_i%5E2">이 되도록 HC0에 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7BN%7D%7BN-K%7D">을 나누어서(1보다 크며, n이 커짐에 따라 1로 수렴하는 term 추가) 스케일링합니다. 즉, 식은 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BHC1%7D%20=%20%5Cfrac%7BN%7D%7BN-K%7D%20%5Ccdot%20%5Coperatorname%7BHC0%7D%20%5C%5C%0A=%20%5Cfrac%7BN%7D%7BN-K%7D%20%5Ccdot%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7Bdiag%7D(e_i%5E2)%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
</section><section id="hc2-leverage-보정" class="level4"><h4 class="anchored" data-anchor-id="hc2-leverage-보정">HC2: Leverage 보정</h4>
<p>위 HC1에서 <img src="https://latex.codecogs.com/png.latex?E(e_i%5E2)%20=%20%5Csigma_i%5E2(1-h_%7Bii%7D)"> 라고 이야기 하였습니다. 일단 그렇다고 하면, 기존에는 <img src="https://latex.codecogs.com/png.latex?%5Csigma_i%5E2">를 <img src="https://latex.codecogs.com/png.latex?e_i%5E2">으로 추정하였다면, 사실 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Be_i%5E2%7D%7B1-h_%7Bii%7D%7D">의 기대값이 <img src="https://latex.codecogs.com/png.latex?%5Csigma_i%5E2">이기 때문에 HC1처럼 앞에 scalar term을 추가하는 대신, HC2는 이를 직접적으로 반영하여 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5CPhi%7D">를 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7Bdiag%7D(e_i%5E2)"> 대신 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7Bdiag%7D(%20%5Cfrac%7Be_i%5E2%7D%7B1-h_%7Bii%7D%7D)">로 추정합니다. 이 때 <img src="https://latex.codecogs.com/png.latex?h_%7Bii%7D">은 <strong>Leverage</strong>(래버리지), <img src="https://latex.codecogs.com/png.latex?H(or%20%5C;h)">는 Hat Matrix라고 불리고, 식은 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BH%7D%20=%20%5Cmathbf%7BX%7D(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%5Cmathbf%7BX%7D%5E%5Ctop">이며, 위 <img src="https://latex.codecogs.com/png.latex?h_%7Bii%7D">은 이 matrix의 대각원소를 뜻합니다. Leverage에 대한 이야기 또한 길지만 간단하게 설명하면, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BH%7D">가 Hat Matrix라고 부르는 이유는 이 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BH%7D"> term에 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7By%7D">를 곱하면, <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BHy%7D%20=%20%5Cmathbf%7BX%7D(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7By%7D%20%5C%5C%0A=%20%5Cmathbf%7BX%7D%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%0A=%5Chat%7B%5Cmathbf%7By%7D%7D%0A"> 이 되어 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7By%7D">에 hat을(추정치) 씌운다는 의미에서 비롯되었으며, 이는 기하학적으로 더욱 복잡하게 설명될 수 있지만 간단한 의미는 <strong>각 관측치가 얼마나 모델의 estimation에 영향을 주었는지를 보여주는 matrix</strong>입니다. 이 matrix의 diagonal값인 레버리지를 보며 관측치 하나하나의 영향력을 볼 수 있고, <strong>이 값이 큰</strong> <strong>관측치에 대해서는 아래 식에서처럼 분모를 작게 함으로써 분산을 크게 추정</strong>한다고 직관적으로 생각할 수 있습니다. <img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BHC2%7D%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7Bdiag%7D%5Cleft(%20%5Cfrac%7Be_i%5E2%7D%7B1-h_%7Bii%7D%7D%20%5Cright)%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>넘어가기 전에, 증명 없이 사용해왔던 수식 <img src="https://latex.codecogs.com/png.latex?E(e_i%5E2)%20=%20%5Csigma_i%5E2(1-h_%7Bii%7D)"> 를 간단하게만 증명해보겠습니다. 예측 오차는 실제값 <img src="https://latex.codecogs.com/png.latex?y_i">와 예측값 <img src="https://latex.codecogs.com/png.latex?%5Chat%7By%7D_i">의 차이입니다: <img src="https://latex.codecogs.com/png.latex?%0Ae_i%20=%20y_i%20-%20%5Chat%7By%7D_i.%0A"> 잔차 벡터 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Be%7D">는 다음과 같이 표현됩니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7Be%7D%20=%20%5Cmathbf%7By%7D%20-%20%5Chat%7B%5Cmathbf%7By%7D%7D%20=%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D)%20%5Cmathbf%7By%7D.%0A"></p>
<p>에러 제곱의 기댓값은 잔차의 분산을 의미합니다. 잔차 벡터 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Be%7D">의 공분산 행렬은 다음과 같이 계산됩니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BCov%7D(%5Cmathbf%7Be%7D)%20=%20%5Ctext%7BCov%7D((%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D)%20%5Cmathbf%7By%7D)%20=%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D)%20%5Ctext%7BCov%7D(%5Cmathbf%7By%7D)%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D)%5ET.%0A"> <img src="https://latex.codecogs.com/png.latex?%5Ctext%7BCov%7D(%5Cmathbf%7By%7D)%20=%20%5Csigma_i%5E2%20%5Cmathbf%7BI%7D">이므로 (당연히 <img src="https://latex.codecogs.com/png.latex?%5Csigma">는 관측치 <img src="https://latex.codecogs.com/png.latex?i">마다 다를 수 있습니다.): <img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BCov%7D(%5Cmathbf%7Be%7D)%20=%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D)%20%5Csigma_i%5E2%20%5Cmathbf%7BI%7D%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D)%5ET%20=%20%5Csigma_i%5E2%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D).%0A"> 여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BH%7D">는 대칭 행렬(<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BH%7D%20=%20%5Cmathbf%7BH%7D%5ET">)이고 멱등 행렬(<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BH%7D%5E2%20=%20%5Cmathbf%7BH%7D">)이므로 (이 부분은 기하학적인 설명이 많이 필요해서 증명하지 않겠다만, 이들은 단지 특정 행렬들의 성질이며 이를 만족한다고 하고 넘기겠습니다.): <img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BCov%7D(%5Cmathbf%7Be%7D)%20=%20%5Csigma_i%5E2%20(%5Cmathbf%7BI%7D%20-%20%5Cmathbf%7BH%7D).%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?i">번째 잔차 <img src="https://latex.codecogs.com/png.latex?e_i">의 분산은 공분산 행렬의 <img src="https://latex.codecogs.com/png.latex?i">번째 대각 성분이고, <img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BVar%7D(e_i)%20=%20%5B%5Ctext%7BCov%7D(%5Cmathbf%7Be%7D)%5D_%7Bii%7D%20=%20%5Csigma_i%5E2%20(1%20-%20h_%7Bii%7D).%0A"> 에러 제곱의 기댓값은 잔차의 분산과 동일하므로 (에러의 mean이 0이므로), <img src="https://latex.codecogs.com/png.latex?%0AE%5Be_i%5E2%5D%20=%20%5Ctext%7BVar%7D(e_i)%20=%20%5Csigma_i%5E2%20(1%20-%20h_%7Bii%7D).%0A"> 가 증명됩니다.</p>
</section><section id="hc3-강화된-leverage-보정" class="level4"><h4 class="anchored" data-anchor-id="hc3-강화된-leverage-보정">HC3: 강화된 Leverage 보정</h4>
<p>HC3은 <strong>Jackknife 접근법</strong>에서 영감을 받아 HC2에서 <img src="https://latex.codecogs.com/png.latex?%5CPhi">의 분모 term을 더 강하게 <img src="https://latex.codecogs.com/png.latex?(1-h_%7Bii%7D)%5E2">로 둡니다. 직관적으로, 같은 <img src="https://latex.codecogs.com/png.latex?h_%7Bii%7D">에 대해 더 크게 분산을 추정 (<img src="https://latex.codecogs.com/png.latex?(1-h_%7Bii%7D)%5E2%20%3C%201-h_%7Bii%7D">)하여, 고레버리지, 즉 outlier에 대해 더 안정적으로 소표본의 분산을 추정할 수 있습니다. <img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BHC3%7D%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7Bdiag%7D%5Cleft(%20%5Cfrac%7Be_i%5E2%7D%7B(1-h_%7Bii%7D)%5E2%7D%20%5Cright)%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
</section><section id="hccmes-비교" class="level4"><h4 class="anchored" data-anchor-id="hccmes-비교">HC(CME)s 비교</h4>
<p>위 내용을 간단하게 표로 정리하면 다음과 같습니다.</p>
<table class="caption-top table">
<colgroup>
<col style="width: 19%">
<col style="width: 38%">
<col style="width: 19%">
<col style="width: 22%">
</colgroup>
<thead><tr class="header">
<th>유형</th>
<th>수식</th>
<th>보정 요소</th>
<th>사용 시나리오</th>
</tr></thead>
<tbody>
<tr class="odd">
<td>HC0</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7Bdiag%7D(e_i%5E2)"></td>
<td>없음</td>
<td>대표본</td>
</tr>
<tr class="even">
<td>HC1</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Bn%7D%7Bn-k%7D%5Coperatorname%7BHC0%7D"></td>
<td>자유도</td>
<td>소표본 기본</td>
</tr>
<tr class="odd">
<td>HC2</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Be_i%5E2%7D%7B1-h_%7Bii%7D%7D"></td>
<td>레버리지 1차</td>
<td>고레버리지 존재 시</td>
</tr>
<tr class="even">
<td>HC3</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Be_i%5E2%7D%7B(1-h_%7Bii%7D)%5E2%7D"></td>
<td>레버리지 2차</td>
<td>소표본 + 고레버리지</td>
</tr>
</tbody>
</table>
<p>정리하자면, 동분산성 검정을 검정하여 이분산성이 확인되면 HC se 시리즈의 적용을 고려해야 하고, 표본 크기가 <img src="https://latex.codecogs.com/png.latex?n%20%3C%2050">인 경우는 HC3 또는 아래에서 다룰 bootstrap기반의 (Clustered) Bootstrap Covariance Matrix Estimation를, <img src="https://latex.codecogs.com/png.latex?n%20%3E%20500">인 경우는 HC0/HC1로 충분하다고 간단하게 생각해볼 수 있습니다. (물론 이는 정해진 규칙이 아니고, 시작은 항상 HC0부터 검정해보는 것이 맞습니다.)<br></p>
</section></section></section><section id="wild-bootstrap" class="level2"><h2 class="anchored" data-anchor-id="wild-bootstrap">5. (Wild) Bootstrap</h2>
<p>마지막으로, sandwich estimator는 아니지만, 모든 모델에서 data로부터 안정적으로 분산을 추정하여 의학 연구에서 인정하는 기법 중 하나인 Bootstraping을 통한 분산 term 추정 기법을 간단하게 소개드리겠습니다. 이 method는 특정 가정으로부터 수학적인 추정식을 세워 분산을 추정하는 대신, Bootstraping 통계 기법을 사용하여 data 만으로 경험적인 분산을 추정합니다. 때문에 이번에 소개하는 (Wild) Bootstrap 은 어떤 가정이나 모델이든 사용할 수 있는, 심지어 2장에서 다룰 clustered(grouped) data 에서도 사용할 수 있는 방법입니다.(사용 대상인 모델이 정해지지 않은, 데이터 만으로 구하는 분산이기 때문에 가정이 전혀 들어가지 않아서입니다. 예를 들어 LM에서 HC se를 사용하는 경우 나머지 세 가정이 만족해야 했지만, Bootstrap은 그렇지 않습니다.) 이후에 다룰 <strong>샌드위치 추정량(Sandwich Estimator)</strong>과, 그의 OLS 버전인 <strong>HC(Heteroskedasticity-Consistent)</strong> 시리즈가 개발되었지만, 이들은 결국 대표본에서의 점근적 성질로부터 얻은 수식이고, 극단적인 <strong>쇼포본</strong> 같은 작은 클러스터 수에서는 이러한 데이터 만을 고려하여 추정하는 부트스트랩이 더 강력한 대안이 될 수도 있습니다. 이를 소개하기 전에 간단히 중요한 통계학적 resampling 기법인 잭나이프(Jackknife)와 부트스트랩(Bootstrap)을 얘기해보겠습니다. 이 둘은 모두 표본을 계속해서 얻기 힘든 상황 (현실에서는 대부분 그럴 것입니다.)에서 모수를 추정하기 위해, 이미 갖고 있는 표본의 data로부터 resampling을 하여 추정하는 통계학적 방법론입니다.</p>
<section id="잭나이프jackknife" class="level4"><h4 class="anchored" data-anchor-id="잭나이프jackknife">잭나이프(Jackknife)</h4>
<p><strong>Jackknife</strong> method는 한 번에 하나의 클러스터를 제외하고 모델을 추정 하는 것을 모든 클러스터에 대해 반복하여 분산을 계산합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathrm%7BVar%7D(%5Chat%7B%5Cbeta%7D_c)%20=%20%5Cfrac%7Bn-1%7D%7Bn%7D%20%5Csum_%7Bc=1%7D%5EC%20%5Cleft(%5Chat%7B%5Cbeta%7D_c%20-%20%5Cbar%7B%5Chat%7B%5Cbeta%7D%7D%5Cright)%5E2,%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D_c">는 c번째 클러스터를 제외한 추정치, <img src="https://latex.codecogs.com/png.latex?%5Cbar%7B%5Chat%7B%5Cbeta%7D%7D">는 추정치의 평균이고, 이는 resampling 기법이지만 위에서 설명드렸던 HC3 추정량이 이 식으로부터 도출된 결과입니다. 물론, 이는 이론적인 결과이므로 각각이 서로 다른 결과를 도출할 수도 있을 것입니다. 단, resampling 기법들은 그 특성상 클러스터나 그 data point의 수가 많을 경우 계산 부하가 큽니다.</p>
</section><section id="부트스트랩bootstrap" class="level4"><h4 class="anchored" data-anchor-id="부트스트랩bootstrap">부트스트랩(Bootstrap)</h4>
<p><strong>Bootstrap</strong> method는 데이터를 무작위로 resampling하여 여러 버전의 데이터셋을 생성하고, 각각에서 모델을 추정한 후 추정값의 변동성을 분산으로 사용합니다. 대표적인 예시로 <strong>와일드 부트스트랩 (Wild Bootstrap)</strong>은 잔차(residual)에 랜덤 가중치를 곱해 인위적 으로 데이터의 분산을 크게 보정하고, <strong>케이스 기반 (XY/Pairs 부트스트랩)</strong>은 기본적인 방법으로 클러스터 자체를 복원 추출하여 새 data를 생성한 뒤 구하는 과정을 반복하며, <strong>잔차 기반 (Residual 부트스트랩)</strong>은 잔차 만을 리샘플링하고 재추정하는 방법입니다다.</p>
</section><section id="wild-bootstrap-1" class="level4"><h4 class="anchored" data-anchor-id="wild-bootstrap-1">Wild Bootstrap</h4>
<p>위에서 설명드렸듯, 잔차 <img src="https://latex.codecogs.com/png.latex?e_i">에 “랜덤 가중치” <img src="https://latex.codecogs.com/png.latex?w_i">를 곱해 새로운 반응 변수를 생성하는 방법으로 다음과 같을 것입니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ay%5E*%20=%20%5Chat%7By%7D%20+%20w_i%20%5Chat%7Be%7D.%0A"></p>
<p>이때 <strong>가중치 분포</strong> 로 사용되는 대표적인 예시는 Rademacher: <img src="https://latex.codecogs.com/png.latex?w_i%20%5Cin%20%5C%7B-1,%201%5C%7D">, Mammen: <img src="https://latex.codecogs.com/png.latex?w_i%20=%20%5Cfrac%7B%5Csqrt%7B5%7D%20%5Cpm%201%7D%7B2%7D">, Webb: 6점 분포(<img src="https://latex.codecogs.com/png.latex?w_i%20%5Cin%20%5Cleft%5C%7B%20-%5Csqrt%7B%5Cfrac%7B3%7D%7B2%7D%7D,%20-%5Csqrt%7B%5Cfrac%7B2%7D%7B2%7D%7D,%20-%5Csqrt%7B%5Cfrac%7B1%7D%7B2%7D%7D,%20%5Csqrt%7B%5Cfrac%7B1%7D%7B2%7D%7D,%20%5Csqrt%7B%5Cfrac%7B2%7D%7B2%7D%7D,%20%5Csqrt%7B%5Cfrac%7B3%7D%7B2%7D%7D%20%5Cright%5C%7D">)로 클러스터별 정규분포 등 다른 적용도 가능합니다. 위에서 언급하였듯, 이외에도 <strong>케이스 기반 (XY) 부트스트랩</strong>, <strong>잔차 부트스트랩</strong>, <strong>프랙셔널 부트스트랩(Fractional Bootstrap)</strong>등의 방법론이 있지만, data를 크게 왜곡하지 않는 Wild가 이러한 분산 추정에서 디폴트하게 고려됩니다.</p>
</section></section><section id="r-예시-hc03-및-부트스트랩" class="level2"><h2 class="anchored" data-anchor-id="r-예시-hc03-및-부트스트랩">6. R 예시: HC0~3 및 부트스트랩</h2>
<p>아래 R 코드를 복사하여 로컬 환경에서 돌려보세요. 위 내용 중 “시리즈가 클 수록 강건하게 추정한다”는 점을 기억하시고 해석하면 됩니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## mtcars 데이터에서 mpg(연비)를 wt(차량 무게), hp(마력)으로 설명하는 회귀모형</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#data(mtcars)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#model &lt;- lm(mpg ~ wt + hp, data = mtcars)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 기본 OLS 표준오차 (등분산 가정)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#summary(model)  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## HC 표준오차 계산</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(sandwich)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(lmtest)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## HC 유형별 분산-공분산 행렬</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cov_hc0 &lt;- vcovHC(model, type = "HC0")  # 기본 White 추정량</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cov_hc1 &lt;- vcovHC(model, type = "HC1")  # 자유도 보정</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cov_hc2 &lt;- vcovHC(model, type = "HC2")  # 잔차 스케일링</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cov_hc3 &lt;- vcovHC(model, type = "HC3")  # 작은 표본 보정</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 계수 테스트 결과 비교</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#coeftest(model, vcov = cov_hc0)  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#coeftest(model, vcov = cov_hc1)  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 부트스트랩 분산-공분산 행렬 (기본 100회 복제)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cov_wild &lt;- vcovBS(fit, cluster = ~cluster_id, type = "wild", R = 1000)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cov_xy &lt;- vcovBS(model, R = 200, type = "xy")  # xy-쌍 부트스트랩</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 결과 비교</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#coeftest(model, vcov = cov_wild)  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#coeftest(model, vcov = cov_xy)  </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 주의: 부트스트랩 결과는 실행 횟수 R, 실행 시마다 다를 수 있음</span></span></code></pre></div></div>
</div>
</section><section id="마무리하며" class="level2"><h2 class="anchored" data-anchor-id="마무리하며">마무리하며</h2>
<p>이번 1장에서는 <strong>Regression Analysis</strong>의 기본 개념과 <strong>(simple, multiple or General) Linear Regression</strong>의 이론적 배경, 그리고 등분산성 가정이 깨진(Heteroskedasticity) 상황에서 유용하게 쓰이는 <strong>Heteroskedasticity-Consistent Standard Errors(HC standard errors)</strong>에 대해 살펴보았습니다.</p>
<p><strong>HC standard errors</strong>를 사용하면, Linear Regression 모델에서 등분산성 가정이 위배되더라도 Standard Errors(or Covariance Matrix)를 좀 더 타당하게 추정할 수 있습니다. 하지만, “<strong>오차항의 독립성</strong>, <strong>선형성</strong>, <strong>정규성</strong>” 등 나머지 가정이 크게 어긋난다면, 단순히 HC SE로 분산을 보정하는 것만으로는 해결되지 않습니다. HC SE가 기본 OLS 분산추정치와 크게 다르다면, “정말 모델이 맞는지”를 다시 고민하고, 필요하다면 모델을 재설정하거나 다른 방법을 모색해야 합니다.</p>
<p>어쨌든, Heteroskedasticity가 의심되는 상황에서 가장 먼저 고려할 만한 접근인 <strong>HC(Heteroskedasticity-Consistent) standard errors</strong>는 모델의 유의성 검정을 위해 안정적으로 분산을 추정할 때 널리 쓰이며, 의학 연구에서도 일상적으로 활용되고 있습니다. 다음 2장에서는 이런 Linear Regression을 더 확장한 <strong>Generalized Linear Model(GLM)</strong>의 개념을 본격적으로 다루고, HC standard errors의 <strong>clustered data</strong> 버전인 <strong>Cluster-robust standard error</strong>에 대해서도 다룰 예정입니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{seungjun2025,
  author = {Seungjun, Lee},
  title = {Exploring {Regression} {Models} for {Regression} {Analysis}
    (1): {Regression} {Analysis,} {Linear} {Regression,} {HC} {Standard}
    {Errors}},
  date = {2025-02-28},
  url = {https://blog.zarathu.com/posts/2025-02-28-reg1/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-seungjun2025" class="csl-entry quarto-appendix-citeas">
Seungjun, Lee. 2025. <span>“Exploring Regression Models for Regression
Analysis (1): Regression Analysis, Linear Regression, HC Standard
Errors.”</span> February 28, 2025. <a href="https://blog.zarathu.com/posts/2025-02-28-reg1/">https://blog.zarathu.com/posts/2025-02-28-reg1/</a>.
</div></div></section></div> ]]></description>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-02-28-reg1/</guid>
  <pubDate>Fri, 28 Feb 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-02-28-reg1/img/reg1.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Exploring Regression Models for Regression Analysis (3): GEE, GLMM, M-statistics, Robust (sandwich) estimation</title>
  <dc:creator>Lee Seungjun</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-02-28-reg3/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="들어가며" class="level2"><h2 class="anchored" data-anchor-id="들어가며">들어가며</h2>
<p>3장에서는 2장에서 다룬 <strong>Generalized Linear Model (GLM)</strong>에서 더 나아가, 데이터 내에 <strong>군집(Clustered)</strong> 구조가 존재하거나, <strong>반복측정(Repeated measures)</strong> 데이터로 인해 <strong>독립성 가정이 깨지는 경우</strong>를 다루는 방법론인 <strong>GEE (Generalized Estimating Equation)</strong>와 <strong>GLMM (Generalized Linear Mixed Model)</strong>을 다루며, 그 전에 중요한 추정 방법론 중 하나인 <strong>M-estimation</strong>과 <strong>Robust(Sandwich) estimation</strong>에 대해서 다루겠습니다.<br></p>
</section><section id="m-estimation" class="level2"><h2 class="anchored" data-anchor-id="m-estimation">1. M-estimation</h2>
<section id="m-estimation-정의" class="level3"><h3 class="anchored" data-anchor-id="m-estimation-정의">1.1. M-estimation 정의</h3>
<hr>
<p><strong>통계 분석</strong>에서 <strong>통계 모델</strong>이 비모수(non-parametric)가 아니라 <strong>모수(parametric)</strong>인 경우, 우리는 model의 모수, 즉 parameter (<img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Ctheta%7D">, <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D"> 등)를 추정해야 합니다. 이는 생각보다 어려운 일이 될 수 있으며, 이전에 언급한 MLE, OLS, Method of Moments (MOM) 등 다양한 model의 estimation 방법이 제안되어 왔습니다. 그런데, 이러한 추정 방법들은 사실상 하나의 <strong>추정방정식(estimating equation, 통계 모델의 parameter 추정 방향을 제시하는 모든 식)</strong>을 세운 뒤, 그 방정식을 만족하는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D">를 찾는 과정으로 해석할 수 있습니다. 예를 들어, <strong>MLE에서는</strong> log likelihood를 parameter로 미분한 함수(score function)가 0이 되는 parameter point를 추정하는 과정이었고, <strong>OLS</strong>에서는 cost function (SSR, 오차 제곱합)을 parameter로 미분한 함수가 0이 되는 parameter point을 찾는 과정이었습니다. <strong>M-estimation은</strong> 이러한 공통된 개념을 일반화하여 공통된 parameter의 성질을 제시해줍니다.</p>
<p>즉, <strong>M-estimation</strong>은 다음 과 같은 형태를 지닌 추정 방정식을 세우고, 이를 만족하는 모수의 값을 해로 삼습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Cboldsymbol%7B%5Ctheta%7D)%20=%20%5Cmathbf%7B0%7D,%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Cpsi_i(%5Cboldsymbol%7B%5Ctheta%7D)">는 (i)번째 관측치에 대해 정의된 <strong>estimating function (e.g.&nbsp;score function)</strong>, <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Ctheta%7D">는 추정하고자 하는 parameter(위 식에서는 우항이 scalar 0이 아닌 벡터 0이므로 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Ctheta%7D"> 또한 벡터.)입니다. 위에서 언급한 것처럼 MLS와 OLS, OLS의 일반화 버전인 Non-linear Least Squares 모두 <strong>M-estimation입니다.</strong> 1, 2장에 걸쳐 이미 익숙하시겠지만 다시 확인해보면, MLE를 M-estimation 형태로 작성해보면 다음과 같고,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cpsi_i(%5Cboldsymbol%7B%5Ctheta%7D)%20=%20%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Ctheta%7D%7D%20%5Clog%20f(Y_i;%20%5Cboldsymbol%7B%5Ctheta%7D)%20%5Cquad%5CLongrightarrow%5Cquad%20%5Csum_%7Bi=1%7D%5En%20%5Cpsi_i(%5Cboldsymbol%7B%5Ctheta%7D)%20=%20%5Csum_%7Bi=1%7D%5En%20%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Ctheta%7D%7D%20%5Clog%20f(Y_i;%20%5Cboldsymbol%7B%5Ctheta%7D)%20=%20%5Cmathbf%7B0%7D.%0A"></p>
<p>OLS는 다음과 같습니다. <img src="https://latex.codecogs.com/png.latex?%0A%5Cpsi_i(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20(Y_i%20-%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cbeta%7D)%20%5Cmathbf%7Bx%7D_i,%20%5Cquad%5CLongrightarrow%5Cquad%20%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7Bx%7D_i(Y_i%20-%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Cmathbf%7B0%7D.%0A"></p>
<p>여기서 OLS의 estimation equation이 다음과 같은 이유는,<br><img src="https://latex.codecogs.com/png.latex?%0A%5Cnabla_%5Cbeta%20J(%5Cbeta)%20=%20%5Cnabla_%5Cbeta%20%5Cfrac%7B1%7D%7B2%7D%20(X%5Cbeta%20-%20y)%5E%5Ctop%20(X%5Cbeta%20-%20y)%20%5C%5C%0A=%20%5Cfrac%7B1%7D%7B2%7D%20%5Cnabla_%5Cbeta%20%5Cbig(%20(X%5Cbeta)%5E%5Ctop%20X%5Cbeta%20-%20(X%5Cbeta)%5E%5Ctop%20y%20-%20y%5E%5Ctop%20(X%5Cbeta)%20+%20y%5E%5Ctop%20y%20%5Cbig)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cfrac%7B1%7D%7B2%7D%20%5Cbig(%202%20X%5E%5Ctop%20X%20%5Cbeta%20-%202%20X%5E%5Ctop%20y%20%5Cbig)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20X%5E%5Ctop%20X%20%5Cbeta%20-%20X%5E%5Ctop%20y%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7Bx%7D_i(%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cbeta%7D%20-%20Y_i)%20=%20%5Cmathbf%7B0%7D%0A"> 에서 유도된 식입니다.</p>
</section><section id="m-estimation-특징" class="level3"><h3 class="anchored" data-anchor-id="m-estimation-특징">1.2. M-estimation 특징</h3>
<hr>
<p><strong>M-estimation</strong>의 가장 중요한 특징은 <strong>일반성과 확장성</strong>입니다. 즉, parameter estimation 문제를</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Cboldsymbol%7B%5Ctheta%7D)%20=%20%5Cmathbf%7B0%7D%0A"></p>
<p>의 해로서 바라보면, 여러 기존 추정법들을 하나의 큰 이론적 틀에서 이해할 수 있고, 이로부터 발생하는 성질들은 해당되는 방법론 모두에 적용됩니다. 이렇게 M-estimation을 강조하여 설명하는 이유는, M-estimation은 아래 두 가지 <strong>수렴 이론(Asymptotic theory)</strong>을 제공하기 때문입니다.</p>
<p>(1) 적절한 <strong>정규성 조건</strong>(regularity conditions) 하에서(종속변수의 정규 분포 가정이 아니며, 언급드린 적이 없지만 아주 general한 조건이라고 생각해주시면 됩니다.), 위 M-estimation의 estimating equation의 추정해 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D">가 참 모수 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Ctheta%7D_0">에 대해 <strong>일치성(Consistency)</strong>과 <strong>점근정규성(Asymptotic Normality)</strong>을 가집니다.</p>
<p><br>
(2) 또한, 정규성을 갖는 모수의 점근분포가 <strong>중심극한정리</strong>(CLT)의 연장선상에 있다고 볼 수 있으며, 그 결과 위 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D">의 asymptotic Normality에서 parameter의 분산은 Asymptotically&amp;robust하게 추정 가능하고, 이 Robust Estimator의 형태가 <strong>샌드위치(sandwich)</strong> 형태로 생겼기 때문에 Sandwich Estimation(or)이라고도 부릅니다.</p>
<p>즉, M-estimation으로부터 얻는 의의를 살펴보자면, 우리가 Regression Model의 parameters를 추정하는 과정에서 estimating equation이 위 M-estimation의 형태를 만족한다면, 어떠한 methods를 사용하든 이를 통해 추정한 parameter <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D">는 참 모수 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Ctheta%7D_0">에 대해 consistent함과, robust한 parameter의 분산을 얻을 수 있다는 것입니다. 이제 (1)과 (2)에 대한 수학적 증명을 걸친 뒤, 이들의 의미를 살펴보겠습니다.</p>
</section><section id="m-estimation의-asymptotic-normality-증명" class="level3"><h3 class="anchored" data-anchor-id="m-estimation의-asymptotic-normality-증명">1.3. M-estimation의 Asymptotic Normality 증명</h3>
<hr>
<p><strong>M-estimation</strong> 추정량 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Ctheta%7D">의 점근적 성질을 유도하기 위해 <strong>1차 Taylor 전개</strong>를 사용합니다. 아래와 같은 M-estimation 추정 방정식 (증명의 편리를 위해 양변에 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B1%7D%7Bn%7D">을 나누었으며, 나누지 않아도 똑같이 증명 가능하고, 등식에서 상수 term을 곱하고 나누는 것은 당연히 문제되지 않습니다. ) <img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Chat%7B%5Ctheta%7D)%20=%200%0A"> 을 참 모수 <img src="https://latex.codecogs.com/png.latex?%5Ctheta_0"> Taylor 식으로 전개하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Chat%7B%5Ctheta%7D)%20%5Capprox%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0)%20+%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20(%5Chat%7B%5Ctheta%7D%20-%20%5Ctheta_0)%20=%200.%0A"></p>
<p>가 됩니다. 이때 좌항 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Chat%7B%5Ctheta%7D)">은 우리가 위 estimating equation에서 보았듯이, 이 항이 0이 되도록 하는 parameter를 추정한 결과가 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Ctheta%7D">였기 때문에 당연히 0일 것이고, 따라서 중앙항 (<img src="https://latex.codecogs.com/png.latex?%5Ctheta_0">에 대한 Taylor 1차 식 전개) 또한 0이 되는 것입니다. 이제 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">에 대한 식을 도출하기 위해 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0)"> term을 넘기고 양변에 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D">을 inverse하여(Matrix 이므로) 곱해주고 <img src="https://latex.codecogs.com/png.latex?%5Csqrt%7Bn%7D">을 곱하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csqrt%7Bn%7D(%5Chat%7B%5Ctheta%7D%20-%20%5Ctheta_0)%20%5Capprox%20-%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cright)%5E%7B-1%7D%20%5Ccdot%20%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0).%0A"></p>
<p>가 됩니다. 여기서 다음 두 Matrix들을 정의하겠습니다:</p>
<ul>
<li>
<img src="https://latex.codecogs.com/png.latex?%20%5Cmathbf%7BA%7D%20=%20%5Cmathbb%7BE%7D%0A%5Cleft%5B%20-%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cright%5D%0A"> (2차 도함수 또는 score function의 미분의 기댓값)</li>
<li>
<img src="https://latex.codecogs.com/png.latex?%20%5Cmathbf%7BB%7D%20=%20%5Cmathbb%7BE%7D%0A%5Cleft%5B%20%5Cpsi_i(%5Ctheta_0)%20%5Cpsi_i(%5Ctheta_0)%5ET%20%5Cright%5D%20"> (score function의 분산의 기댓값)</li>
</ul>
<p>이제 <strong>대수의 법칙(LLN)</strong>과 <strong>중심극한정리(CLT)</strong>를 각각 적용하면 다음 두 식을 얻을 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A-%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cxrightarrow%7Bp%7D%20%5Cmathbf%7BA%7D%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0)%20%5Cxrightarrow%7Bd%7D%20%5Cmathcal%7BN%7D(0,%20%5Cmathbf%7BB%7D)%0A"></p>
<p>각 정리를 간단하게 설명드리자면, <strong>대수의 법칙(LLN, Law of Large Numbers)</strong>은 표본 크기 <img src="https://latex.codecogs.com/png.latex?n">이 충분히 크면, 표본 평균이 모평균에 점근적으로 수렴한다는 법칙으로, 확률 변수 <img src="https://latex.codecogs.com/png.latex?X_i">가 동일 분포이고 기대값 <img src="https://latex.codecogs.com/png.latex?%5Cmathbb%7BE%7D%5BX_i%5D%20=%20%5Cmu">를 가지면, 수학적으로 표현하면 아래 식과 같습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20X_i%20%5Cxrightarrow%7Bp%7D%20%5Cmu.%0A"></p>
<p>즉, 위 식에서는 <img src="https://latex.codecogs.com/png.latex?%0A-%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%0A%5Capprox%20-%20%5Cmathbb%7BE%7D%5Cleft%5B%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cright%5D%0A=%20%5Cmathbf%7BA%7D%0A"> 가 되는 것입니다. <strong>중심극한정리(CLT, Central Limit Theorem)</strong>는 독립이고 동일한 분포를 따르는 확률변수들의 표본 평균이 정규 분포를 따른다는 정리로, 분산이 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">인 확률변수 <img src="https://latex.codecogs.com/png.latex?X_i">들에 대해,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20(X_i%20-%20%5Cmu)%20%5Cxrightarrow%7Bd%7D%20%5Cmathcal%7BN%7D(0,%20%5Csigma%5E2).%0A"> 입니다. (양변에 <img src="https://latex.codecogs.com/png.latex?%5Csqrt%7Bn%7D">이 나눠진 식이 더 친숙하실 겁니다.) 즉, 위 식에서는 <img src="https://latex.codecogs.com/png.latex?%5Cpsi_i(%5Ctheta_0)=0">이고, 따라서 <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BB%7D%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20%5Cpsi_i(%5Ctheta_0)%20%5Cpsi_i(%5Ctheta_0)%5ET%20%5Cright%5D%20=%20Var(%5Cpsi_i(%5Ctheta_0))%0A"> 이므로,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0)%20%5Cxrightarrow%7Bd%7D%20%5Cmathcal%7BN%7D(0,%20%5Cmathbf%7BB%7D)%0A"> 임을 확인할 수 있습니다. 정리하자면, 대수의 법칙이 평균값으로의 수렴을 보장한다면, 중심극한정리는 표본 평균이 정규성을 띤다는 것을 보장하고, 이를 통해 우리가 고려하던 아래 식 <img src="https://latex.codecogs.com/png.latex?%0A%5Csqrt%7Bn%7D(%5Chat%7B%5Ctheta%7D%20-%20%5Ctheta_0)%20%5Capprox%20-%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cright)%5E%7B-1%7D%20%5Ccdot%20%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0).%0A"> 의 우항에 대한 두 정보를 얻을 수 있었습니다. 최종적으로 점근정규성을 보이기 위해선 이 두 수렴하는 분포의 곱을 나타낼 수 있는 <strong>Slutsky 정리</strong>를 보고, 최종적으로 식을 도출하겠습니다. <strong>Slutsky 정리</strong>는 두 개의 점근적 확률 분포를 결합하는 방법으로, 만약 <img src="https://latex.codecogs.com/png.latex?X_n%20%5Cxrightarrow%7Bd%7D%20X"> (약한 수렴)과, <img src="https://latex.codecogs.com/png.latex?Y_n%20%5Cxrightarrow%7Bp%7D%20c"> (확률적 수렴)이면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AX_i%20Y_i%20%5Cxrightarrow%7Bd%7D%20Xc.%0A"></p>
<p>입니다. 즉, 확률적으로 수렴하는 변수와 분포적으로 수렴하는 변수를 곱하면, 여전히 위 식과 같이 분포적으로 수렴한다는 것이 증명된 정리이고, 위 식에서는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cxrightarrow%7Bp%7D%20A%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0)%20%5Cxrightarrow%7Bd%7D%20%5Cmathcal%7BN%7D(0,%20B)%0A"> 이므로, Slutsky 정리를 사용하면 최종적으로</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csqrt%7Bn%7D(%5Chat%7B%5Ctheta%7D%20-%20%5Ctheta_0)%20%5Capprox%20-%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cright)%5E%7B-1%7D%20%5Ccdot%20%5Cfrac%7B1%7D%7B%5Csqrt%7Bn%7D%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Ctheta_0).%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cxrightarrow%7Bd%7D%20A%5E%7B-1%7D%20%5Cmathcal%7BN%7D(0,%20B)%20=%20%5Cmathcal%7BN%7D(0,%20A%5E%7B-1%7D%20B%20A%5E%7B-1%7D)%0A"></p>
<p>입니다.(deteminant한 값은 분산 term에서 제곱된다는 것은 몇 번 보았었습니다.) 결국 M-estimation의 추정을 통해 얻은 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D">가 참 모수 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Ctheta%7D_0">에 대해 일치성(Consistency)을 갖고, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D">는 점근정규성(Asymptotic Normality)을 갖으며 그 식은 <img src="https://latex.codecogs.com/png.latex?%5Cmathcal%7BN%7D(0,%20A%5E%7B-1%7D%20B%20A%5E%7B-1%7D)">입니다. 또한, 이 샌드위치(sandwich) 형태(<img src="https://latex.codecogs.com/png.latex?A%5E%7B-1%7D"> 빵 사이에 껴있는 고기 <img src="https://latex.codecogs.com/png.latex?B">)처럼 생긴 점근적 분산 식이 바로 Sandwich estimator의 general version입니다. 이제 이 sandwich estimator에 대해 좀더 설명드리겠습니다.</p>
</section><section id="sandwichrobust-estimator" class="level3"><h3 class="anchored" data-anchor-id="sandwichrobust-estimator">1.4. Sandwich(Robust) Estimator</h3>
<hr>
<p>Sandwich(Robust) Estimator의 식은 써보면 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Coperatorname%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Ctheta%7D%7D)%20=%20A%5E%7B-1%7D%20B%20A%5E%7B-1%7D%0A"> <img src="https://latex.codecogs.com/png.latex?%0Awhere,%20%5C;%20%5Cmathbf%7BA%7D%20=%20-%20%5Cmathbb%7BE%7D%5Cleft%5B%20%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Ctheta_0)%7D%7B%5Cpartial%20%5Ctheta%5ET%7D%20%5Cright%5D,%0A%5C;%20%5Cmathbf%7BB%7D%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20%5Cpsi_i(%5Ctheta_0)%20%5Cpsi_i(%5Ctheta_0)%5ET%20%5Cright%5D%0A"></p>
<p>2장에서 GLM case에 대해 log likelihood의 1차 도함수를 score function, 이의 negative 2차 도함수를 Fisher Information matrix라고 언급한 적이 있습니다. 이의 general한 버전이 위와 같으며, 여기에서는 이 score function <img src="https://latex.codecogs.com/png.latex?%5Cpsi_i(%5Ctheta_0)">의 분산을 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BB%7D">, 2차 도함수를 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D">로 표기하고 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D">는 모형의 <strong>곡률(curvature)</strong>을, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BB%7D">은 모형의 분산을 반영합니다. 또한, Estimating equation이 log likelihood로부터 MLE 철학으로 나온 parameter라면, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D%20=%20%5Cmathbf%7BB%7D">입니다. 이 이유는, 2장에서 2차 도함수가 정의되는 임의의 distribution을 따르는 종속변수 <img src="https://latex.codecogs.com/png.latex?Y">와 그의 parameter <img src="https://latex.codecogs.com/png.latex?%5Ctheta">에 대해서 <img src="https://latex.codecogs.com/png.latex?%20%5Cell''%20=%20%5Cfrac%7Bd%5E2%5Cell%7D%7Bd%5Ctheta%5E2%7D%20"> 가 참임을 보였고, <img src="https://latex.codecogs.com/png.latex?%5Cell''">은 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D">, <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Bd%5E2%5Cell%7D%7Bd%5Ctheta%5E2%7D">는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BB%7D">와 같기 때문입니다. (<img src="https://latex.codecogs.com/png.latex?%5Cell'%20=%20%5Cpsi"> 이므로.)</p>
<p>즉 철학적으로 해석해보면, Regression Model의 selection이 정확한 경우 Fisher Information 행렬 동일성에 의해 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D%20=%20%5Cmathbf%7BB%7D">가 성립하게 되고, 이에 따라 <strong>parameter의 분산은</strong> <img src="https://latex.codecogs.com/png.latex?A%5E%7B-1%7D"> <strong>만으로 추정</strong>될 수 있습니다. 그러나 <strong>Regression Model이 정확하지 않은 경우, consistent한 parameter estimation을 한다고 하더라도 이 모델의 추정 분산</strong> <img src="https://latex.codecogs.com/png.latex?A%5E%7B-1%7D"><strong>은 더이상 신뢰할 수 없으며, 이때 Sandwich estimtor는 경험적 분산(empirical variance)</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BB%7D"><strong>를 통해 robust하게 이를 추정할 수 있습니다.</strong> 즉, Regression Model의 몇 가지 가정이 의심될 때, 심지어는 의심되지 않더라도 Sandwich estimtor는 robust하게 parameter의 분산을 추정할 수 있는 것입니다.</p>
<p>또한 이전에 스포한대로, 이전 장들에서 다루어 왔던 robust한 parameter variance estimator인 Heteroskedasticity-Consistent SE, Cluster-robust SE는 모두 이 Sandwich estimator의 special한 case입니다.(생김새부터 짐작할 수 있으셨을 겁니다.) LM version에서만 이를 증명한 뒤(GLM 버전도 같습니다.), GLM을 복습하고 GEE, GLMM에 대해서 설명드리겠습니다.</p>
<ol type="1">
<li><strong>Prove HC0 is Sandwich estimator. (LM version)</strong></li>
</ol>
<p>OLS의 score function은은 위에서 보았듯 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cpsi_i(%5Cbeta)%20=%20x_i%20(Y_i%20-%20x_i%5ET%20%5Cbeta).%0A"> 그렇다면, <img src="https://latex.codecogs.com/png.latex?A">는 베타로 미분 후 -를 씌워주면 다음과 같이 계산되며,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AA%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20-%5Cfrac%7B%5Cpartial%20%5Cpsi_i(%5Cbeta)%7D%7B%5Cpartial%20%5Cbeta%5ET%7D%20%5Cright%5D%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20x_i%20x_i%5ET%20%5Cright%5D.%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?A">의 추정치는 결국</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7BA%7D%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20x_i%20x_i%5ET%20=%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20X.%0A"> 가 됩니다. 마찬가지로 <img src="https://latex.codecogs.com/png.latex?B">를 계산하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AB%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20%5Cpsi_i(%5Cbeta)%20%5Cpsi_i(%5Cbeta)%5ET%20%5Cright%5D%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20x_i%20x_i%5ET%20(Y_i%20-%20x_i%5ET%20%5Cbeta)%5E2%20%5Cright%5D.%0A"></p>
<p>이며, 추정치는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7BB%7D%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20x_i%20x_i%5ET%20e_i%5E2%20=%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20%5Ctext%7Bdiag%7D(e_i%5E2)%20X.%0A"> 입니다. 결국</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D(%5Chat%7B%5Cbeta%7D)%20=%20%5Chat%7BA%7D%5E%7B-1%7D%20%5Chat%7BB%7D%20%5Chat%7BA%7D%5E%7B-1%7D%20=%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20X%20%5Cright)%5E%7B-1%7D%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20%5Ctext%7Bdiag%7D(e_i%5E2)%20X%20%5Cright)%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20X%20%5Cright)%5E%7B-1%7D.%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ET%20X)%5E%7B-1%7D%20X%5ET%20%5Ctext%7Bdiag%7D(e_i%5E2)%20X%20(X%5ET%20X)%5E%7B-1%7D.%0A"></p>
<p>가 되고, 이 식은 1장에서 보았던 <strong>HC0</strong>의 식과 동일함을 확인할 수 있습니다.</p>
<ol start="2" type="1">
<li><strong>Prove Clustered-Robust SE is Sandwich estimator. (LM version)</strong></li>
</ol>
<p>이 또한 OLS와 같은 환경이므로(LM, cluster가 <img src="https://latex.codecogs.com/png.latex?g">개로 구성되어 있다고 할 때, score function은 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cpsi_g(%5Cbeta)%20=%20%5Csum_%7Bi%20%5Cin%20g%7D%20x_i%20(Y_i%20-%20x_i%5ET%20%5Cbeta).%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?A">의 식과 추정치 또한 비슷하게 구해지고,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AA%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20-%5Cfrac%7B%5Cpartial%20%5Cpsi_g(%5Cbeta)%7D%7B%5Cpartial%20%5Cbeta%5ET%7D%20%5Cright%5D%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20%5Csum_%7Bi%20%5Cin%20g%7D%20x_i%20x_i%5ET%20%5Cright%5D.%0A"> <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7BA%7D%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20%5Csum_%7Bi%20%5Cin%20g%7D%20x_i%20x_i%5ET%20=%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20X.%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?B">도 비슷하게 계산되며, cluster간의 independent는 여전히 가정됩니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AB%20=%20%5Cmathbb%7BE%7D%20%5Cleft%5B%20%5Cpsi_g(%5Cbeta)%20%5Cpsi_g(%5Cbeta)%5ET%20%5Cright%5D.%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7BB%7D%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20%5Cleft(%20%5Csum_%7Bi%20%5Cin%20g%7D%20x_i%20e_i%20%5Cright)%20%5Cleft(%20%5Csum_%7Bi%20%5Cin%20g%7D%20x_i%20e_i%20%5Cright)%5ET%20=%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20X_g%5ET%20%5Chat%7Bu%7D_g%20%5Chat%7Bu%7D_g%5ET%20X_g.%0A"></p>
<p>이에 따라 분산의 Sandwich estimator를 구하면</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D(%5Chat%7B%5Cbeta%7D)%20=%20%5Chat%7BA%7D%5E%7B-1%7D%20%5Chat%7BB%7D%20%5Chat%7BA%7D%5E%7B-1%7D%20=%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20X%20%5Cright)%5E%7B-1%7D%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20X_g%5ET%20%5Chat%7Bu%7D_g%20%5Chat%7Bu%7D_g%5ET%20X_g%20%5Cright)%20%5Cleft(%20%5Cfrac%7B1%7D%7Bn%7D%20X%5ET%20X%20%5Cright)%5E%7B-1%7D.%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20(X%5ET%20X)%5E%7B-1%7D%20%5Cleft(%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20X_g%5ET%20%5Chat%7Bu%7D_g%20%5Chat%7Bu%7D_g%5ET%20X_g%20%5Cright)%20(X%5ET%20X)%5E%7B-1%7D.%0A"></p>
<p>이고, 이는 Cluster-robust SE의 식과 동일합니다.</p>
</section><section id="glm-복습" class="level3"><h3 class="anchored" data-anchor-id="glm-복습">1.5. GLM 복습</h3>
<hr>
<p><strong>Generalized Linear Model (GLM)</strong>의 모델 식은 다음과 같이 표현됩니다: <img src="https://latex.codecogs.com/png.latex?%0Ag(%5Cmathbb%7BE%7D%5BY_i%20%7C%20X_i%5D)%20=%20g(%5Cmu_i)%20=%5Ceta_i%20=%20X_i%5ET%20%5Cbeta%20%5C%5C%0Awhere,%20Y_i%20%5Csim%20%5Ctext%7BExponential%20Family%7D(%5Cmu_i,%20%5Cphi).%0A"> 이때 <img src="https://latex.codecogs.com/png.latex?g(%5Ccdot)">은 링크 함수(link function)로 logit, log의 예시를 보았고, <img src="https://latex.codecogs.com/png.latex?%5Cmu_i%20=%20%5Cmathbb%7BE%7D%5BY_i%20%7C%20X_i%5D">는 반응 변수의 기대값으로 모델의 mapping의 목적이 되는 값(예측하고자 하는 값), <img src="https://latex.codecogs.com/png.latex?%5Cphi"> 분산과 관련된 parameter(dispersion parameter) 로 정규 분포의 경우 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">였습니다.</p>
<p>간략하게 복습하면 <strong>링크 함수(link function)</strong> <img src="https://latex.codecogs.com/png.latex?g(%5Ccdot)">를 통해 <img src="https://latex.codecogs.com/png.latex?E(Y_i)%20=%20%5Cmu_i">와 <img src="https://latex.codecogs.com/png.latex?%5Ceta_i%20=%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cboldsymbol%7B%5Cbeta%7D">를 연결하고, <strong>분산 함수(variance function)</strong> <img src="https://latex.codecogs.com/png.latex?V(%5Cmu_i)">를 이용해 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D(Y_i)">를 표현하며, <strong>추정방정식(estimating equation)</strong>을 세워</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csum_%7Bi=1%7D%5En%20%5Cfrac%7B%20%5Cpartial%20%5Cmu_i%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%5E%5Ctop%7D%20%5Cfrac%7B%20(Y_i%20-%20%5Cmu_i)%20%7D%7B%5Coperatorname%7BVar%7D(Y_i)%7D%20=%20%5Cmathbf%7B0%7D%0A"></p>
<p>을 푸는 방식으로 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">를 구합니다.</p>
<p>이 해석 또한 M-estimation의 한 사례로 볼 수 있습니다. GLM에서 score 함수(추정방정식)는 <img src="https://latex.codecogs.com/png.latex?%5Cpsi_i(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Cfrac%7B%20%5Cpartial%20%5Cmu_i%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%5E%5Ctop%7D%20%5Cfrac%7B%20(Y_i%20-%20%5Cmu_i)%20%7D%7B%5Coperatorname%7BVar%7D(Y_i)%7D"> 꼴로 정의되며, 이를 0으로 만드는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">가 <strong>우리가 구하고자 하는 파라미터 추정치</strong>가 됩니다. 2장에서는 GLM의 parameter <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">를 추정하는 방법으로 IRLS(Iteratively Reweighted Least Squares)이나 Newton-Raphson/Fisher Scoring을 소개하였으며, 이는 결국 M-estimation에서 구체적으로 어떻게 “estimating equation을 수치적으로 풀어낼지” 알고리즘으로 구현한 예시 중에 하나였다고 이해할 수 있습니다. 또한, 2장에서 예고한대로, 왜 parameter <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">의 분산이</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AVar(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20-fisher%0A"> 라고 했었는지 이제 살펴보면, GLM의 추정 또한 M-estimation에 해당하므로 GLM의 estimating equation을 만족하는 estimator에 대해서 위에서 확인한 점근정규성이 만족함을 알 수 있고, 때문에 consistent한 parameter estimator에 대해서 다음과 같은 robust한 Sandwich 분산을 갖습니다.<br><img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%20-%20%5Cboldsymbol%7B%5Cbeta%7D_0)%20%5Capprox%20%5Cleft(%20%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D_0)%20%5Cright)%5E%7B-1%7D%20%5Coperatorname%7BVar%7D(%5Cmathbf%7BS%7D(%5Cboldsymbol%7B%5Cbeta%7D_0))%20%5Cleft(%20%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D_0)%20%5Cright)%5E%7B-1%7D%0A"></p>
<p>그리고, 계속 보아왔던 것처럼 여기서 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D(%5Cmathbf%7BS%7D(%5Cboldsymbol%7B%5Cbeta%7D_0))%20=%20%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D_0)">가 만족(스코어 함수의 분산의 기댓값과 Fisher information matrix가 같습니다.) 하기 때문에 이를 대입하면 위 식이 아래 처럼 소거되고,<br><br><img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%20-%20%5Cboldsymbol%7B%5Cbeta%7D_0)%20%5Capprox%20%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D_0)%5E%7B-1%7D%0A"></p>
<p>가 됩니다. 결국 GLM의 <strong>모형 기반 분산</strong>은 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cmathbb%7BV%7Dar_%7B%5Ctext%7B%EB%AA%A8%ED%98%95%7D%7D(%5Chat%7B%5Cbeta%7D)%20=%20%5Cmathbf%7BA%7D%5E%7B-1%7D%20=%20%5Cleft(%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7B%5Cpartial%5E2%20%5Clog%20f(Y_i;%20%5Cbeta)%7D%7B%5Cpartial%20%5Cbeta%20%5Cpartial%20%5Cbeta%5ET%7D%20%5Cright)%5E%7B-1%7D.%20"></p>
<p>또한, 이때 경험적 분포를 고려하여 Sandwich로 추정한 분산은,</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cmathbb%7BV%7Dar_%7B%5Ctext%7Brobust%7D%7D(%5Chat%7B%5Cbeta%7D)%20=%20%5Cmathbf%7BA%7D%5E%7B-1%7D%20%5Cmathbf%7BB%7D%20%5Cmathbf%7BA%7D%5E%7B-1%7D,%20"></p>
<p><img src="https://latex.codecogs.com/png.latex?%20where,%20%5Cmathbf%7BB%7D%20=%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cpsi_i(%5Chat%7B%5Cbeta%7D)%20%5Cpsi_i(%5Chat%7B%5Cbeta%7D)%5ET,%20%5C;%20%5Cpsi_i(%5Cbeta)%20=%20(Y_i%20-%20%5Cmu_i)%20x_i%20/%20V(%5Cmu_i).%20"></p>
<p>로, 이는 이전에 확인한 HC0의 형태와도 이어집니다.</p>
</section><section id="다시-돌아와서..-for-clustered-data" class="level3"><h3 class="anchored" data-anchor-id="다시-돌아와서..-for-clustered-data">다시 돌아와서.. (for clustered data)</h3>
<p>일반적으로 <strong>선형 모델</strong>(Linear Model)과 <strong>일반화 선형 모델</strong>(Generalized Linear Model, GLM)은 <strong>독립 동일 분포(i.i.d.)</strong>를 가정합니다. 즉, 기존의 GLM은 관측치(observations, data points)들이 서로 독립이며(Independent)일 때 동일한 분산 구조에서 잘 작동합니다. 그러나 학교나 병원 등 군집(클러스터) 단위로 샘플이 묶여 있는, 비슷한 특성을 지닌 대상들을 <strong>클러스터(cluster)</strong>로 묶은 <strong>패널 데이터(panel data)</strong>나 동일한 실험 대상(피험자)에게서 <strong>반복 측정된 데이터(longitudinal data)</strong>의 경우, 같은 cluster(또는 group: 같은 피험자, 같은 단위 등)에 속한 data간에는 correlation이 존재합니다. 때문에 더이상 data들이 독립이 아니게 되며, GLM만으로는 이 상관구조를 모델 자체에서 고려할 수 없기에, <strong>GEE</strong>와 <strong>GLMM</strong> 와 같은, 더욱 general한 Regression Model이 개발되었습니다. 이제 아래에서 위 두 model에 대해서 살펴보겠습니다.<br></p>
</section></section><section id="generalized-estimating-equation-gee" class="level2"><h2 class="anchored" data-anchor-id="generalized-estimating-equation-gee">2. Generalized Estimating Equation (GEE)</h2>
<section id="gee-정의" class="level3"><h3 class="anchored" data-anchor-id="gee-정의">2.1. GEE 정의</h3>
<hr>
<p><strong>GEE (Generalized Estimating Equation)</strong>는 <strong>GLM</strong>이 독립성 가정을 전제로 하는 한계마저 뛰어넘어, <strong>군집(Clustered) 자료</strong>나 <strong>반복측정 자료</strong> 등 <strong>상관구조</strong>가 존재하는 데이터에 적용될 수 있도록 확장한 방법론입니다. 가장 critical하게 다른 점을 보면, GLM은 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D(Y_i)%20=%20%5Cphi%20V(%5Cmu_i)">로 종속변수의 분산을 표현할 때 diagonal matrix로 두어 data points 간에는 correlation이 없음을 표현하였다면, GEE는 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D(%5Cmathbf%7BY%7D_i)%20=%20%5Cphi%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D%5Cmathbf%7BR%7D_i(%5Calpha)%5Cmathbf%7BA%7D_i%5E%7B1/2%7D">와 같은 <strong>working correlation</strong> 행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">를 사용하여, 관측치들 간의 <strong>상관관계</strong>를(반복 측정, 클러스터 내 상관) 모델에 반영합니다. 또한, 이러한 가정을 위해 종속변수의 확률 모델(공동 확률 분포)을 완전히 명시하지 않아도, <strong>Quasi-Likelihood(준 우도) 접근법</strong>을 통해 <strong>점근적(score) 방정식</strong>을 확장하여 모델을 적합합니다. 간단히 말하면, GEE는 “<strong>평균 모형</strong>은 GLM처럼 유지하되, <strong>상관 구조</strong>를 적절히 지정하여 <strong>군집성</strong>이나 <strong>반복측정</strong>을 고려하자”라는 접근입니다.</p>
<p><strong>LM</strong>이나 <strong>GLM</strong>은 서로 독립적인(i.i.d.) 표본을 가정하여 이를 기반으로 추정하는 반면, GEE에서는 <strong>상관 구조(correlation structure)</strong> <img src="https://latex.codecogs.com/png.latex?R(%5Calpha)">를 추가하여 이러한 독립 가정을 완화하고, 평균 모형과 분산-공분산 구조에 대한 가정을 분리해서 Quasi 형태로 추정합니다. 이 때, Quasi-likelihood에 대한 적용을 짧게 설명하자면, <strong>GEE는 확률 모델을 직접 설정(완전한 공동 확률 밀도 함수 명시)하지 않고, GLM의 log likelihood function에 상관구조를 추가하는 형태로 접근</strong>합니다. 즉 GLM에서 <em>종속 변수의 Exponential family 가정 -&gt; 독립 가정 후 모든 data point의 확률(likelihood)를 곱해서 얻은 likelihood finction -&gt; 로그 씌워서 log likelihood -&gt; model parameter로 미분한 결과인 score function</em> 순으로 추정 과정을 설명했었다면, GEE는 <em>처음부터 직접적인 종속변수의 가정으로 시작하는 대신 log likelihood에서 시작</em>하고, 이 때 독립이 아님을 고려하기 위해 variance function에 상관 행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">을 추가하여 <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7BVar%7D(%5Cmathbf%7BY%7D_i)%20=%20%5Cphi%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D%5Cmathbf%7BR%7D_i(%5Calpha)%5Cmathbf%7BA%7D_i%5E%7B1/2%7D">로 둔 후 추정하는 것입니다. 이러한 접근은, 실제로 종속변수의 완전한 확률적 기반(joint PDF)이 존재하지 않아도, 점근적 성질을 활용하여 <strong>일관된 추정량</strong>을 얻을 수 있고, <strong>오류 항이 독립적이지 않은 경우</strong>에도 GLM과 유사한 방식으로 추정할 수 있는 장점이 있습니다. 마지막으로, GEE는 상관행렬과 Quasi의 개념을 통해 GLM과 같이 data points들을 marginal하게 고려하여 fit하기 때문에 <strong>Population-Average GEE(or 모델)</strong> 이고, 이는 무작위 효과(Random Effect)를 통해 각 실험 단위(피험자)에 특화된 효과를 추정하는 G<strong>LMM(Generalized Linear Mixed Model)</strong>과 철학이 다르며,이 GLMM은 <strong>Subject-Specific GEE(or 모델)</strong>이라고도 부릅니다.</p>
</section><section id="gee-수학적-표현-및-추정" class="level3"><h3 class="anchored" data-anchor-id="gee-수학적-표현-및-추정">2.2. GEE 수학적 표현 및 추정</h3>
<hr>
<p>위에서 언급하였듯, GLM과 동일하게 GEE는 아래와 같은 marginal 모델입니다:<br><img src="https://latex.codecogs.com/png.latex?%0Ag%5Cbigl(%5Cboldsymbol%7B%5Cmu%7D_i%5Cbigr)%20=%20%5Cmathbf%7BX%7D_i%20%5Cboldsymbol%7B%5Cbeta%7D,%0A"> 여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">는 (i)번째 클러스터(또는 피험자)에서 나온 <img src="https://latex.codecogs.com/png.latex?n_i">개의 관측치 벡터, <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cmu%7D_i%20=%20E(%5Cmathbf%7BY%7D_i)%20%5C;%20or%20%5C;%20E(%5Cmathbf%7BY%7D_i%7CX_i)">입니다. 또한, 언급한 대로 Working correlation을 아래와 같이 설정하며,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(%5Cmathbf%7BY%7D_i)%20=%20%5Cphi%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D%20%5Cmathbf%7BR%7D_i(%5Calpha)%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D.%0A"> 이때 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D_i">가 기존 <img src="https://latex.codecogs.com/png.latex?V(%5Cmu_%7Bij%7D)">의 역할 이었다면 이에 루트를 씌워 A라고 두고 (행렬에서의 square root, 또는 1/2 승은 기존 <img src="https://latex.codecogs.com/png.latex?V">가 diagonal 이었으므로 이때는 단순히 각 대각 성분을 루트 씌운 값입니다.) 그 사이에 클러스터 당 상관관계를 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">로 표현합니다. 이때 상관행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D(%5Calpha)">의 종류로는 크게 아래와 같은 예시들이 있습니다.</p>
<p>(1) Independent (기존 GLM)</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AR(%5Calpha)%20=%20I,%20%5Cquad%20V_k%20=%20V_k'%0A"></p>
<p>(2) Exchangeable Correlation (동일 상관 구조)</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AR(%5Calpha)%20=%20%5Cbegin%7Bpmatrix%7D%0A1%20&amp;%20%5Calpha%20&amp;%20%5Calpha%20%5C%5C%0A%5Calpha%20&amp;%20%5Cddots%20&amp;%20%5Calpha%20%5C%5C%0A%5Calpha%20&amp;%20%5Calpha%20&amp;%201%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p>(3) Autoregressive (AR-1)</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BCorr%7D(y_%7Bki%7D,%20y_%7Bkj%7D)%20=%20%5Calpha%5E%7B%7Ci-j%7C%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AR(%5Calpha)%20=%20%5Cbegin%7Bpmatrix%7D%0A1%20&amp;%20%5Calpha%20&amp;%20%5Calpha%5E2%20&amp;%20%5Cdots%20&amp;%20%5Calpha%5E%7Bn_k%7D%20%5C%5C%0A%5Calpha%20&amp;%201%20&amp;%20%5Calpha%20&amp;%20%5Cdots%20&amp;%20%5Calpha%5E%7Bn_k-1%7D%20%5C%5C%0A%5Cvdots%20&amp;%20%5Cvdots%20&amp;%20%5Cvdots%20&amp;%20%5Cddots%20&amp;%20%5Cvdots%20%5C%5C%0A%5Calpha%5E%7Bn_k%7D%20&amp;%20%5Calpha%5E%7Bn_k-1%7D%20&amp;%20%5Calpha%5E%7Bn_k-2%7D%20&amp;%20%5Cdots%20&amp;%201%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p>(4) Unstructured Form</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AR(%5Calpha)%20=%20%5Cbegin%7Bpmatrix%7D%0A1%20&amp;%20%5Calpha_1%20&amp;%20%5Calpha_2%20&amp;%20%5Calpha_3%20%5C%5C%0A%5Calpha_1%20&amp;%201%20&amp;%20%5Calpha_4%20&amp;%20%5Calpha_5%20%5C%5C%0A%5Calpha_2%20&amp;%20%5Calpha_4%20&amp;%201%20&amp;%20%5Calpha_6%20%5C%5C%0A%5Calpha_3%20&amp;%20%5Calpha_5%20&amp;%20%5Calpha_6%20&amp;%201%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p>이러한 상관행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D(%5Calpha)">는 사전에 정의되어야 하므로 분석 대상인 data의 성질에 따라 선정해야 하며, 이러한 관계의 구조를 어떻게 선택하는 지에 따라 분산이 다르게 나오므로, 위에서 다룬 Sandwich를 통한 robust한 추정이 GEE에서 대게 사용됩니다.</p>
<section id="gees-estimating-equation" class="level4"><h4 class="anchored" data-anchor-id="gees-estimating-equation">GEE’s Estimating Equation</h4>
<p>이전에 GLM에서는 다음과 같이 score functions로 부터 estimating equation을 세웠습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CPsi%20=%20%5Csum%20%5Cpsi_i%20=%20%5Csum_%7Bi=1%7D%5E%7BN%7D%20%5Cfrac%7By_i%20-%20%5Cmu_i%7D%7Ba(%5Cphi)V(%5Cmu_i)%7D%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu_i%7D%7B%5Cpartial%20%5Ceta_i%7D%20%5Cright)%20x_i%20=%200%0A"></p>
<p>이제 이를 GLM 때와 다르게 각 cluster <img src="https://latex.codecogs.com/png.latex?k">에 대해 <img src="https://latex.codecogs.com/png.latex?D_k%20=%20diag(%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu_i%7D%7B%5Cpartial%20%5Ceta_i%7D%20%5Cright)%20x_i)">, <img src="https://latex.codecogs.com/png.latex?V_k%20=%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D%20%5Cmathbf%7BR%7D_i(%5Calpha)%20%5Cmathbf%7BA%7D_i%5E%7B1/2%7D">라고 하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csum_%7Bk=1%7D%5E%7BK%7D%20%5Cfrac%7B1%7D%7Ba(%5Cphi)%7D%20%20D_k%20V_k%5E%7B-1%7D%20(y_k%20-%20%5Cmu_k)%20=%200%0A"> 으로 식을 GLM의 score function 으로부터 변형해서 얻을 수 있고, 최종적으로 벡터와 행렬 연산으로 모든 클러스터 <img src="https://latex.codecogs.com/png.latex?k">에 대해 block diagonal로 한 번에 표현하면(<img src="https://latex.codecogs.com/png.latex?V">), <img src="https://latex.codecogs.com/png.latex?%0A%5CPsi%20=%20%5Cfrac%7B1%7D%7Ba(%5Cphi)%7D%20D%20V%5E%7B-1%7D%20(y%20-%20%5Cmu)%20=%200%0A"> 가 되며, <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BU%7D(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20%5Cbigl(%5Cmathbf%7BY%7D_i%20-%20%5Cboldsymbol%7B%5Cmu%7D_i%5Cbigr)%20=%20%5Cmathbf%7B0%7D,%0A"> 로 표현할 수 있습니다. 이는 GLM의 score 함수와 같은 시작이지만, GEE는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">가 군집/반복측정 상관을 반영하도록 조작합니다. 결국 이 추정방정식을 품으로써 GEE의 추정이 가능할 것입니다.</p>
<p>가장 중요한 GEE에서 분산 term의 변형을 다시 한 번 강조하자면, <img src="https://latex.codecogs.com/png.latex?V_k"> 는 cluster 별 (Co)variance 행렬로, data가 independent하다면 <img src="https://latex.codecogs.com/png.latex?V_k">와 이를 모두 합친 <img src="https://latex.codecogs.com/png.latex?V">가 diagonal matrix가 되지만, 그룹 내 상관을 고려할 경우 <img src="https://latex.codecogs.com/png.latex?V_k">가 더 이상 diagonal하지 않고, 이에 따라 <img src="https://latex.codecogs.com/png.latex?V">는 block diagonal matrix 형태를 갖습니다. (block diagonal한 이유는 cluster끼리 독립이고 cluster안은 상관관계가 있는 1차 clustered data에서 다룬 2장의 cluster-robust를 떠올리면 좋을 것 같습니다.)</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AV%20=%20%5Cbegin%7Bpmatrix%7D%0AV_1%20&amp;%200%20&amp;%200%20%5C%5C%0A0%20&amp;%20%5Cddots%20&amp;%200%20%5C%5C%0A0%20&amp;%200%20&amp;%20V_K%0A%5Cend%7Bpmatrix%7D%0A"></p>
</section><section id="gee-parameter-추정irls" class="level4"><h4 class="anchored" data-anchor-id="gee-parameter-추정irls">GEE parameter 추정(IRLS)</h4>
<p>GEE의 parameter 추정 또한 GLM에서 비롯된 만큼, 이전에 다루었던 방식과 유사한 반복 알고리즘으로 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">를 추정할 수 있습니다. 하나의 스텝을 예시로 들어보면,</p>
<ol type="1">
<li><p>현재 추정치 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t)%7D">에서, 각 클러스터 <img src="https://latex.codecogs.com/png.latex?i">에 대해 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BD%7D_i"> (편미분 행렬), <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cmu%7D_i%20=%20%5Cmu(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t)%7D)">, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i"> (working correlation <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">, dispersion parameter <img src="https://latex.codecogs.com/png.latex?%5Cphi"> 포함)을 계산합니다. 이때, <strong>working correlation</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">와 <img src="https://latex.codecogs.com/png.latex?%5Cphi">도 반복적으로 업데이트됩니다. 예컨대, <code>gee</code>나 <code>geepack</code> 패키지에서는 각 반복 단계에서 <strong>잔차(residual)</strong>를 기반으로 <img src="https://latex.codecogs.com/png.latex?%5Calpha">와 <img src="https://latex.codecogs.com/png.latex?%5Cphi">를 재추정하여 새로운 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D_i">를 구합니다.</p></li>
<li>
<p>아래 식을 만족하도록 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t+1)%7D">를 업데이트합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t+1)%7D%0A=%20%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t)%7D%0A-%20%5Cleft(%20%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20%5Cmathbf%7BD%7D_i%20%5Cright)%5E%7B-1%7D%0A%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20%5Cbigl(%5Cmathbf%7BY%7D_i%20-%20%5Cboldsymbol%7B%5Cmu%7D_i%5Cbigr).%0A"></p>
</li>
<li><p>이전처럼 parameter의 변화량(distance between <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t+1)%7D">and <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5E%7B(t)%7D">)가 특정 threshold 아래로 수렴할 때까지 이 과정을 반복합니다.</p></li>
</ol>
<p>R에서는 <code>geepack</code>이나 <code>gee</code> 라이브러리에서 내부적으로 이러한 절차를 수행합니다. 1에서 어떻게 잔차로부터 <img src="https://latex.codecogs.com/png.latex?%5Calpha">를 추정할 수 있는지 간단하게 예시를 보면 아래와 같습니다. 이는 이전과 원리는 같으며, 상관행렬에서 추정해야 하는 parameter 개수에 따른 자유도를 고려하기 때문에 식이 좀더 복잡해진 것입니다.</p>
<p>잔차는 아래와 같이 계산됩니다(Pearson): <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7Br%7D_%7Bki%7D%20=%20%5Cfrac%7By_%7Bki%7D%20-%20%5Chat%7B%5Cmu%7D_%7Bki%7D%7D%7B%5Csqrt%7BV(%5Chat%7B%5Cmu%7D_%7Bki%7D)%7D%7D,%0A"> where <img src="https://latex.codecogs.com/png.latex?y_%7Bki%7D">는 observed response for cluster <img src="https://latex.codecogs.com/png.latex?k"> and observation <img src="https://latex.codecogs.com/png.latex?i">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmu%7D_%7Bki%7D">는 predicted mean for cluster <img src="https://latex.codecogs.com/png.latex?k"> and observation <img src="https://latex.codecogs.com/png.latex?i">, <img src="https://latex.codecogs.com/png.latex?V(%5Chat%7B%5Cmu%7D_%7Bki%7D)">는 variance function evaluated at <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmu%7D_%7Bki%7D">. 이제 <img src="https://latex.codecogs.com/png.latex?n_k%20=%20n">라고 가정하면 다음과 같습니다. (이는 클러스터 당 data point 개수가 같다는 가정이며, 이를 만족하지 않아도 식이 복잡해질 뿐 똑같이 계산됩니다.)</p>
<p><strong>(2) Exchangeable Correlation:</strong> <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Calpha%7D%20=%20%5Cfrac%7B1%7D%7Ba(%5Cphi)%7D%20%5Csum_%7Bk=1%7D%5EK%20%5Csum_%7Bi%20%3E%20j%7D%20%5Cfrac%7B%5Chat%7Br%7D_%7Bki%7D%20%5Chat%7Br%7D_%7Bkj%7D%7D%7BK%20%5Ccdot%20%5Cfrac%7B1%7D%7B2%7Dn(n-1)%20-%20p%7D,%0A"></p>
<p><strong>(3) Autoregressive (AR-1):</strong> <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Calpha%7D%20=%20%5Cfrac%7B1%7D%7Ba(%5Cphi)%7D%20%5Csum_%7Bk=1%7D%5EK%20%5Csum_%7Bi=1%7D%5E%7Bn_k%20-%201%7D%20%5Cfrac%7B%5Chat%7Br%7D_%7Bki%7D%20%5Chat%7Br%7D_%7Bk(i+1)%7D%7D%7BK(n-1)%20-%20p%7D,%0A"></p>
<p><strong>(4) Unstructured Form:</strong> <img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7Ba%7D_%7Bij%7D%20=%20%5Cfrac%7B1%7D%7Ba(%5Cphi)%7D%20%5Csum_%7Bk=1%7D%5EK%20%5Cfrac%7B%5Chat%7Br%7D_%7Bki%7D%20%5Chat%7Br%7D_%7Bkj%7D%7D%7BK%20-%20p%7D,%0A"></p>
<p>위 식들은 그저 잔차로부터 (co)variance를 추정하는 것일 뿐이고, 분산 term은 degree of freedom을 고려하기 때문에 그저 각각의 상관 행렬 속 미지수(parameter)의 개수에 따른 반영입니다.</p>
</section></section><section id="gee-parameters-variance" class="level3"><h3 class="anchored" data-anchor-id="gee-parameters-variance">2.3. GEE parameter’s Variance</h3>
<hr>
<p>GEE의 <strong>모수 추정치</strong> (<img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%5Cbeta%7D">)도 <strong>M-estimation</strong>의 범주에 속하므로, 점근 분산은 <strong>Sandwich</strong> 형태를 갖습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Coperatorname%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%5Cbeta%7D)_%7B%5Ctext%7Brobust%7D%7D%0A=%20%5Cleft(%20%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%0A%5Cmathbf%7BD%7D_i%20%5Cright)%5E%7B-1%7D%20%5Cleft(%20%5Csum_%7Bi=1%7D%5En%0A%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20(%5Cmathbf%7BY%7D_i%20-%0A%5Cboldsymbol%5Cmu_i)%20(%5Cmathbf%7BY%7D_i%20-%20%5Cboldsymbol%5Cmu_i)%5E%5Ctop%0A%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20%5Cmathbf%7BD%7D_i%20%5Cright)%20%5Cleft(%20%5Csum_%7Bi=1%7D%5En%0A%5Cmathbf%7BD%7D_i%5E%5Ctop%20%5Cmathbf%7BV%7D_i%5E%7B-1%7D%20%5Cmathbf%7BD%7D_i%20%5Cright)%5E%7B-1%7D.%0A"></p>
<p>이를 <strong>robust</strong> 또는 <strong>empirical</strong> 표준오차라고 하며, 실질적으로 <strong>상관구조</strong> (<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BR%7D_i(%5Calpha)">)가 부정확하게 지정되었을지라도 일관성을 보장해 줍니다. 즉, 위 M-estimation으로 GLM을 해석할 때와 일치하게, <strong>Model-based SE</strong> 는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D%5E%7B-1%7D">만을 사용해서 계산하는 것이고, 이는 설정한 working correlation(상관행렬) 가정이 정확하다고 믿을 때이기 때문에, 이를 신뢰할 수 없는 경우 거의 무조건 <strong>Robust SE</strong><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D%5E%7B-1%7D%5Cmathbf%7BB%7D%5Cmathbf%7BA%7D%5E%7B-1%7D"> 를 사용합니다. 이는 R에서 GEE를 계산할 때 <strong>기본 SE</strong>(model-based)와 <strong>robust SE</strong>(empirical) 두 가지가 함께 리포팅되는 이유이기도 합니다.<br></p>
</section></section><section id="generalized-linear-mixed-model-glmm" class="level2"><h2 class="anchored" data-anchor-id="generalized-linear-mixed-model-glmm">3. Generalized Linear Mixed Model (GLMM)</h2>
<section id="glmm-정의" class="level3"><h3 class="anchored" data-anchor-id="glmm-정의">3.1. GLMM 정의</h3>
<hr>
<p><strong>GLMM(Generalized Linear Mixed Model)</strong>은, 우리가 이미 익숙하게 다뤄온 <strong>GLM(Generalized Linear Model)</strong>을 GEE와는 다른 방식으로 (Mixed model) “군집(cluster) 또는 계층적 구조를 가지는 자료”에까지 확장하기 위한 방법론입니다. 즉, GLMM은 이러한 <strong>내재된 상관(또는 군집성)</strong>을 모델화하기 위해서 <strong>고정 효과 + 무작위 효과</strong>의 결합으로 모형을 설정합니다. 즉, GLMM은</p>
<ul>
<li>
<strong>고정 효과(fixed effects)</strong>: 전체 모집단에 공통적으로 적용되는 회귀계수(예: 전체 평균 경향)에 해당,</li>
<li>
<strong>무작위 효과(random effects)</strong>: 피험자(또는 군집, 클러스터)별로 달라지는 편차(“개인별 random intercept” 혹은 “개인별 random slope” 등)를 도입</li>
</ul>
<p>을 둘다 고려하는 모델이며, 즉 <strong>“Generalized Linear Model + Linear Mixed Model(Random Effects)”의 결합</strong>이라고 요약할 수 있습니다. GEE와 비교하여 이 GLMM은 각 cluster(또는 group)마다 직접적인 고려를 모델에 넣기 때문에(random effect) <strong>Subject-Specific 모델(또는 GEE)</strong>라고도 불리며, 이는 Population-Average GEE와 대비되는 특징입니다. 무작위 효과는 <strong>정규분포</strong>로 가정하는 것이 일반적이며, 경우에 따라서는 다른 분포(예: Gamma)로 가정하기도 하고, GLMM에서은 이러한 LMM을 GLM으로 확정한 것이기 때문에 종속변수의 분포를, <strong>Exponential Family</strong>로 확장합니다.<br></p>
</section><section id="lmm-수학적-표현-및-추정" class="level3"><h3 class="anchored" data-anchor-id="lmm-수학적-표현-및-추정">3.2. LMM 수학적 표현 및 추정</h3>
<hr>
<p>GLMM을 이해하기 위해서는 먼저 <strong>선형혼합모형(LMM; Linear Mixed Model)</strong>을 확실하게 이해할 필요가 있습니다. (이 LMM과 GLMM을 완벽하게 이전처럼 분석하려면 내용이 산만해지기 때문에 여기선 중요한 점을 위주로 짚고, 추가적인 공부가 필요하신 분들은 위키피디아에서 비롯되는 교재 및 논문 내용들을 집중적으로 살펴보시면 좋을 것 같습니다.) LMM은 종속변수 <img src="https://latex.codecogs.com/png.latex?Y">의 분포가 정규분포라는 전제하에서, <strong>고정 효과(fixed effects)</strong>와 <strong>무작위 효과(random effects)</strong>가 동시에 존재한다고 보는 모형입니다.</p>
<section id="lmm-수학적-표현" class="level4"><h4 class="anchored" data-anchor-id="lmm-수학적-표현">LMM 수학적 표현</h4>
<hr>
<p>가장 단순한 형태의 LMM(임의절편 모형, random intercept model)을 생각해 보겠습니다. (이때 LMM에서 모형을 나누는 기준은 random effect, 즉 group을 어느 정도로 복잡하게 고려하는 지에 따른 설계의 차이입니다. random effect의 분포, 차원 등을 다양하게 고려할 수 있겠지요.) 예를 들어, <img src="https://latex.codecogs.com/png.latex?i">번째 클러스터(또는 피험자) 내에서 <img src="https://latex.codecogs.com/png.latex?j">번째 관측값을 나타내는 <img src="https://latex.codecogs.com/png.latex?Y_%7Bij%7D">를 다음과 같이 모델링합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY_%7Bij%7D%20=%20%5Cbeta_0%20+%20%5Cbeta_1%20X_%7Bij%7D%20+%20b_i%20+%20%5Cvarepsilon_%7Bij%7D,%0A%5Cquad%20i=1,%5Cdots,K,%5Cquad%20j=1,%5Cdots,n_i.%0A"></p>
<p>이때 <img src="https://latex.codecogs.com/png.latex?%5Cbeta_0,%20%5Cbeta_1">은 <strong>고정 효과(fixed effects)</strong> parameter, 즉 모든 클러스터에 공통 적용되는 평균적인 효과)이고, <img src="https://latex.codecogs.com/png.latex?b_i">는 <strong>무작위 효과(random effect)</strong> parameter로, 클러스터 <img src="https://latex.codecogs.com/png.latex?i">마다 서로 다른 절편(intercept) 편차를 갖는 것을 모델링하고 있습니다. <img src="https://latex.codecogs.com/png.latex?%5Cvarepsilon_%7Bij%7D">는 흔히 오차항(residual)으로 간주하고, 대게 <img src="https://latex.codecogs.com/png.latex?%5Cvarepsilon_%7Bij%7D%20%5Csim%20N(0,%20%5Csigma%5E2)">로 가정합니다.</p>
<p>추가적으로, 무작위 효과 <img src="https://latex.codecogs.com/png.latex?b_i">는 다음과 같은 분포로 가정합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ab_i%20%5Csim%20N(0,%5C;%5Ctau%5E2).%0A"></p>
<p>이는 “(피험자마다) 임의로 달라지는 절편(intercept)”이 정규분포를 따른다는 것을 의미합니다. 모든 <img src="https://latex.codecogs.com/png.latex?b_i">를 <strong>독립</strong>으로 가정하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(b_i)%20=%20%5Ctau%5E2,%5Cquad%20%5Coperatorname%7BVar%7D(%5Cvarepsilon_%7Bij%7D)%20=%20%5Csigma%5E2.%0A"></p>
<p>결국, 어떤 <img src="https://latex.codecogs.com/png.latex?Y_%7Bij%7D">에 대해서는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY_%7Bij%7D%20=%20%5Cbeta_0%20+%20b_i%20+%20%5Cbeta_1%20X_%7Bij%7D%20+%20%5Cvarepsilon_%7Bij%7D,%0A"></p>
<p>이고,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(Y_%7Bij%7D)%20=%20%5Ctau%5E2%20+%20%5Csigma%5E2.%0A"></p>
<p>이먀, 더 일반화 된 모델로 무작위 절편 + 무작위 기울기(random intercept + random slope)를 도입하여 독립변수 <img src="https://latex.codecogs.com/png.latex?X">에 대해서도 개인별로 기울기가 달라지도록 만들 수 있습니다. 이 경우,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY_%7Bij%7D%0A=%20(%5Cbeta_0%20+%20b_%7B0i%7D)%20+%20(%5Cbeta_1%20+%20b_%7B1i%7D)%20X_%7Bij%7D%20+%20%5Cvarepsilon_%7Bij%7D,%0A%5Cquad%0Ab_%7B0i%7D%20%5Csim%20N(0,%5C;%5Ctau_%7B00%7D),%5C;%0Ab_%7B1i%7D%20%5Csim%20N(0,%5C;%5Ctau_%7B11%7D),%5C;%0A%5Coperatorname%7BCov%7D(b_%7B0i%7D,%20b_%7B1i%7D)%20=%20%5Ctau_%7B01%7D.%0A"></p>
<p>가 될 것입니다. 이처럼 무작위 효과를 하나 혹은 여러 개 갖는다는 것은, “클러스터마다 고유하게 발생하는 변동”을 모델에 포함하는 방식으로, LMM은 이러한 방식로 <strong>상관구조</strong>를 모델링 해낸다고 생각할 수 있습니다. 이를 벡터와 행렬 형태로 표현해보면, 각 클러스터(또는 피험자) <img src="https://latex.codecogs.com/png.latex?i">에 대해</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BY%7D_i%0A=%20%5Cmathbf%7BX%7D_i%5C,%5Cboldsymbol%7B%5Cbeta%7D%0A%5C;+%5C;%20%5Cmathbf%7BZ%7D_i%5C,%5Cmathbf%7Bb%7D_i%0A%5C;+%5C;%20%5Cboldsymbol%7B%5Cvarepsilon%7D_i,%0A"></p>
<ul>
<li>
<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">: <img src="https://latex.codecogs.com/png.latex?i">번째 클러스터에서의 <img src="https://latex.codecogs.com/png.latex?n_i">차원 응답벡터</li>
<li>
<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D_i">: <img src="https://latex.codecogs.com/png.latex?n_i%20%5Ctimes%20p"> 차원의 <strong>고정 효과 설계 행렬</strong>(fixed effect parameter <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">와 매칭)</li>
<li>
<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BZ%7D_i">: <img src="https://latex.codecogs.com/png.latex?n_i%20%5Ctimes%20q"> 차원의 <strong>무작위 효과 설계 행렬</strong>(random effect parameter <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">와 매칭)</li>
<li>
<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i%20%5Csim%20N(%5Cmathbf%7B0%7D,%20%5Cboldsymbol%7BG%7D)"> 이며 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7BG%7D">는 <img src="https://latex.codecogs.com/png.latex?q%20%5Ctimes%20q"> 공분산 행렬</li>
<li>
<img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cvarepsilon%7D_i%20%5Csim%20N(%5Cmathbf%7B0%7D,%20%5Csigma%5E2%20%5Cmathbf%7BI%7D_%7Bn_i%7D)">로 일반적으로 가정(독립 동일 분포)</li>
</ul>
<p>여기서 설계 행렬이란, 1장의 LM에서부터 사용하였지만, data point(observation)당 미리 input으로 지정되는 행렬로, 정확한 의미는 “일련의 개체에 대한 설명 변수 값을 나열한 행렬로 각 행은 개별 개체를 나타내며, 열은 해당 개체에 대한 변수 및 특정 값에 해당한다”입니다. X는 계속 봐왔지만 Z는 이번에 처음 나온 설계 행렬인데, 이는 각 data point마다 해당되는 cluster에는 1, 해당되지 않는 나머지 cluster는 0의 값을 갖는, cluster를 선택하는 스위치 느낌으로, input으로 정해지는 행렬이라고 생각하시면 됩니다.</p>
<p>이 LMM의 (Co) variance matrix는 단순하게 분산 term을 씌우면 random한 (determinant하지 않은) 항만 남아 다음과 같이 계산 될 것입니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Coperatorname%7BVar%7D(%5Cmathbf%7BY%7D_i)%0A=%20%5Cmathbf%7BZ%7D_i%5C,%5Cboldsymbol%7BG%7D%5C,%5Cmathbf%7BZ%7D_i%5E%5Cmathsf%7BT%7D%0A+%20%5Csigma%5E2%5C,%5Cmathbf%7BI%7D_%7Bn_i%7D.%0A"></p>
</section><section id="lmms-parameter-추정maximum-likelihood-reml" class="level4"><h4 class="anchored" data-anchor-id="lmms-parameter-추정maximum-likelihood-reml">LMM’s parameter 추정(Maximum Likelihood, REML)</h4>
<hr>
<p>이 LMM에서 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">, <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7BG%7D"> (또는 <img src="https://latex.codecogs.com/png.latex?%5Ctau%5E2"> 등), <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2"> 모두 미지수입니다. 이를 추정하기 위해 주로 <strong>최대우도추정법(ML; Maximum Likelihood)</strong> 또는 <strong>제한최대우도추정법(REML; Restricted Maximum Likelihood)</strong>을 사용합니다. 각각이 어떻게 계산될지 설명드리면 다음과 같습니다.</p>
<p><strong>(1) With ML(MLE).</strong><br><img src="https://latex.codecogs.com/png.latex?b_i">가 정규분포라는 가정 하에, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">의 <strong>joint distribution</strong>도 다변량 정규분포로 표현할 수 있습니다. 모든 <img src="https://latex.codecogs.com/png.latex?i">에 대해 <img src="https://latex.codecogs.com/png.latex?b_i"> 또는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">가 독립이라 가정하면(cluster간은 독립), 전체 자료의 joint density를 곱해서 <strong>likelihood 함수</strong>를 정의할 수 있고, 이를 최대화하는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">와 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7BG%7D%7D">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Csigma%7D%5E2">를 찾으면 됩니다. 단점으로는, ML은 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D"> 추정에서 편향(bias)이 발생할 수 있어, 표본 크기가 작거나 모형 구조가 복잡해질 때 문제될 수 있습니다. 수식을 중요 부분만 전개해보면, LMM에서 모든 <img src="https://latex.codecogs.com/png.latex?b_i">를 각각 적분하여(즉 클러스터 마다 적분) 얻은 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">의 분포는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BY%7D_i%20%5Csim%20N(%5Cmathbf%7BX%7D_i%20%5Cbeta,%20V_i)%20%5Cquad%20%5Ctext%7Bwith%7D%20%5Cquad%20V_i%20=%20%5Cmathbf%7BZ%7D_i%20%5Cmathbf%7BG%7D%20%5Cmathbf%7BZ%7D_i%5ET%20+%20%5Csigma%5E2%20I_%7Bn_i%7D.%0A"></p>
<p>입니다. 이제 <img src="https://latex.codecogs.com/png.latex?i">번째 클러스터 자료 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">에 대한 log-likelihood를 계산해보면, 다차원 정규분포이므로 다음과 같이 나옵니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cell_i(%5Cbeta,%20%5Cmathbf%7BG%7D,%20%5Csigma%5E2)%20=%20-%5Cfrac%7B1%7D%7B2%7D%20%5Cleft%5B%20n_i%20%5Clog(2%5Cpi)%20+%20%5Clog%20%7CV_i%7C%20+%20(%5Cmathbf%7BY%7D_i%20-%20%5Cmathbf%7BX%7D_i%20%5Cbeta)%5ET%20V_i%5E%7B-1%7D%20(%5Cmathbf%7BY%7D_i%20-%20%5Cmathbf%7BX%7D_i%20%5Cbeta)%20%5Cright%5D.%0A"></p>
<p>이를 전체 <img src="https://latex.codecogs.com/png.latex?K">개의 cluster에 대해 모두 합하면, 전체 자료에 대한 log-likelihood 함수 <img src="https://latex.codecogs.com/png.latex?%5Cell(%5Cbeta,%20%5Cmathbf%7BG%7D,%20%5Csigma%5E2)">가 도출될 것입니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cell(%5Cbeta,%20%5Cmathbf%7BG%7D,%20%5Csigma%5E2)%20=%20%5Csum_%7Bi=1%7D%5E%7BK%7D%20%5Cell_i(%5Cbeta,%20%5Cmathbf%7BG%7D,%20%5Csigma%5E2).%0A"></p>
<p>MLE(<img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D_%7BML%7D,%20%5Chat%7B%5Cmathbf%7BG%7D%7D_%7BML%7D,%20%5Chat%7B%5Csigma%7D%5E2_%7BML%7D">)를 구하기 위해서는 위의 log-likelihood를 <img src="https://latex.codecogs.com/png.latex?%5Cbeta,%20%5Cmathbf%7BG%7D,%20%5Csigma%5E2">에 대해 미분하여 0이 되게 하는 해를 찾으면 됩니다. 그러나 일반적으로 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D">와 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">에 대한 미분은 해석적으로 단순화하기 어렵고, 또한 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D">가 양의 정부호(positive definite)가 되어야 하는 제약이 있으므로, 수치적 최적화(EM 알고리즘, Newton-Raphson, Fisher scoring 등)를 사용해야 합니다.</p>
<p><strong>(2) With REML.</strong><br>
이는 ML를 직접 바로 계산하는 대신, 고정 효과 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">와 관련이 없는 term을 이용해 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7BG%7D">와 <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2">를 먼저 추정한 후 모델을 추정하는 방식입니다. 일반적으로 LMM 을 추정할 때는 REML이 <strong>고정 효과</strong> 추정에 대한 편의를 줄여주고, 분산 요소에 대한 추정이 좀 더 안정적이기 때문에 ML보다 선호됩니다. 이는 모델에서 <strong>무작위 효과를 적분(marginal likelihood)</strong>하는 접근을 통해 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D,%20%5Cboldsymbol%7BG%7D,%20%5Csigma%5E2">에 대한 우도 함수를 세우고(restricted likeli hood), 이 함수를 최대화하는 방식으로 진행됩니다. 실제 계산은 <strong>Iterative 알고리즘(EM 알고리즘, 또는 Fisher scoring, Newton-Raphson 등)</strong>을 사용합니다. 식을 보면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cell_%7BREML%7D%20(%5Cmathbf%7BG%7D,%20%5Csigma%5E2)%20=%20-%5Cfrac%7B1%7D%7B2%7D%20%5Cleft%5B%20%5Csum_%7Bi=1%7D%5E%7BK%7D%20%5Clog%20%7CV_i%7C%20+%20%5Clog%20%7C%5Cmathbf%7BX%7D%5ET%20%5Cmathbf%7BV%7D%5E%7B-1%7D%20%5Cmathbf%7BX%7D%7C%20+%20(%5Cmathbf%7BY%7D%20-%20%5Cmathbf%7BX%7D%20%5Chat%7B%5Cbeta%7D)%5ET%20%5Cmathbf%7BV%7D%5E%7B-1%7D%20(%5Cmathbf%7BY%7D%20-%20%5Cmathbf%7BX%7D%20%5Chat%7B%5Cbeta%7D)%20%5Cright%5D%20+%20%5Ctext%7Bconst%7D.%0A"></p>
<p>이고, 이때 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BV%7D%20=%20%5Ctext%7Bblockdiag%7D(V_1,%20%5Cdots,%20V_K)">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D">는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D,%20%5Csigma%5E2">가 주어졌을 때의 일반화최소제곱(GLS) 해입니다. 여기서 <img src="https://latex.codecogs.com/png.latex?%5Clog%20%7C%5Cmathbf%7BX%7D%5ET%20%5Cmathbf%7BV%7D%5E%7B-1%7D%20%5Cmathbf%7BX%7D%7C">가 REML에서 추가로 나타나는 항으로, 이것이 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">를 제거(또는 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">에 무관한 부분만 모아놓음)하여 우선적으로 식을 전개한 효과라고 이해할 수 있습니다. REML에서는 이 <img src="https://latex.codecogs.com/png.latex?%5Cell_%7BREML%7D%20(%5Cmathbf%7BG%7D,%20%5Csigma%5E2)">를 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D,%20%5Csigma%5E2">에 대해 최대화한 뒤, 그 해 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BG%7D%7D_%7BREML%7D,%20%5Chat%7B%5Csigma%7D%5E2_%7BREML%7D">를 이용해 최종적으로 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D_%7BREML%7D"> 을 구합니다. REML은 ML보다 fixed effect estimator의 편향 문제가 덜하며, 분산 성분 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BG%7D,%20%5Csigma%5E2">에 대해 좀 더 안정적인 추정을 제공합니다. 특히 샘플이 작거나 무작위효과 구조가 복잡할 때 일반적으로 더욱 안정적인 REML을 권장하는 편입니다.</p>
<p>이를 통해 얻은 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7BG%7D%7D">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Csigma%7D%5E2">는 <strong>LMM의 MLE(or REML) 추정치</strong>이며, R에서는 <code>lme4</code> 패키지 등에서 이 과정을 내부적으로 수행합니다.<br></p>
</section></section><section id="glmm의-수학적-표현-및-추정" class="level3"><h3 class="anchored" data-anchor-id="glmm의-수학적-표현-및-추정">3.3. GLMM의 수학적 표현 및 추정</h3>
<hr>
<p>이제 LMM에서 정규 오차항을 일반화하여, 종속변수가 이항, 포아송, 혹은 다른 지수분포족을 따를 수 있도록 확장하면, <strong>GLMM</strong>으로 이어집니다. GLMM은</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ag%5Cbigl(%5Cmu_%7Bij%7D%5Cbigr)%0A=%20%5Cmathbf%7Bx%7D_%7Bij%7D%5E%5Cmathsf%7BT%7D%5C,%5Cboldsymbol%7B%5Cbeta%7D%0A+%20%5Cmathbf%7Bz%7D_%7Bij%7D%5E%5Cmathsf%7BT%7D%5C,%5Cmathbf%7Bb%7D_i%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0Awhere,%20%5C;%20%5Cmathbf%7Bb%7D_i%20%5Csim%20N(%5Cmathbf%7B0%7D,%20%5Cboldsymbol%7BG%7D),%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AY_%7Bij%7D%20%5Cmid%20%5Cmathbf%7Bb%7D_i%20%5Csim%20%5Ctext%7BExponential%20Family%7D(%5Cmu_%7Bij%7D,%20%5Cphi),%0A"></p>
<p>의 구조입니다. 직관적으로도 GLMM은 LMM+GLM임을 볼 수 있고, 당연히 <img src="https://latex.codecogs.com/png.latex?g(%5Ccdot)">는 link function으로, GLM과 마찬가지로 <img src="https://latex.codecogs.com/png.latex?%5Cmu_%7Bij%7D%20=%20E(Y_%7Bij%7D%20%5Cmid%20%5Cmathbf%7Bb%7D_i)">를 <strong>적절한 링크 함수</strong> <img src="https://latex.codecogs.com/png.latex?g">로 mapping하는 함수이며, 예시로 binomial case에서 로짓 링크(logit)를 사용하면 <img src="https://latex.codecogs.com/png.latex?Y_%7Bij%7D">는 0 또는 1 값을 가지는 이항분포가 될 수 있고, 이는 <img src="https://latex.codecogs.com/png.latex?%5Clog%5Cbigl(%5Cmu_%7Bij%7D/(1-%5Cmu_%7Bij%7D)%5Cbigr)">를 회귀식을 표현하는 것이었습니다.</p>
<p>이때, 위 식의 likelihood는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BY%7D_i%20%5Cmid%20%5Cmathbf%7Bb%7D_i%0A%5Csim%20%5Cprod_%7Bj=1%7D%5E%7Bn_i%7D%20f%5Cbigl(Y_%7Bij%7D%5Cmid%20%5Cmu_%7Bij%7D(%5Cmathbf%7Bb%7D_i)%5Cbigr),%0A"></p>
<p>로 쓸 수 있으며, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">를 적분(marginalizing over <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">)하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ap(%5Cmathbf%7BY%7D_i)%0A=%20%5Cint%0A%5Cprod_%7Bj=1%7D%5E%7Bn_i%7D%0Af%5Cbigl(Y_%7Bij%7D%5Cmid%20%5Cmu_%7Bij%7D(%5Cmathbf%7Bb%7D_i)%5Cbigr)%5C,%0A%5Cvarphi%5Cbigl(%5Cmathbf%7Bb%7D_i%5Cbigr)%5C,%0Ad%5Cmathbf%7Bb%7D_i,%0A"></p>
<p>가 최종적으로 <strong>cluster</strong> <img src="https://latex.codecogs.com/png.latex?i">에 대한 (marginal) 분포를 만들어냅니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ap(%5Cmathbf%7BY%7D_i)%20=%20%5Cint%20%5Cleft%5B%20%5Cprod_%7Bj=1%7D%5E%7Bn_i%7D%20f(Y_%7Bij%7D%20%7C%20%5Cmu_%7Bij%7D%20(b_i),%20%5Cphi)%20%5Cright%5D%20%5Cvarphi%20(b_i)%20db_i,%20%5Cquad%20%5Cvarphi%20(b_i)%20=%20%5Cfrac%7B1%7D%7B%5Csqrt%7B(2%5Cpi)%5Eq%20%7C%5Cmathbf%7BG%7D%7C%7D%7D%20%5Cexp%20%5Cleft(-%5Cfrac%7B1%7D%7B2%7D%20b_i%5ET%20%5Cmathbf%7BG%7D%5E%7B-1%7D%20b_i%20%5Cright).%0A"></p>
<p>모든 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BY%7D_i">가 (조건부) 독립이라면, 전체 자료에 대한 marginal likelihood는</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AL(%5Cbeta,%20%5Cmathbf%7BG%7D,%20%5Cphi)%20=%20%5Cprod_%7Bi=1%7D%5E%7BK%7D%20%5Cint%20%5Cprod_%7Bj=1%7D%5E%7Bn_i%7D%20f(Y_%7Bij%7D%20%7C%20%5Cmu_%7Bij%7D%20(b_i),%20%5Cphi)%20%5Cvarphi%20(b_i)%20db_i.%0A"></p>
<p>입니다. 문제는 <img src="https://latex.codecogs.com/png.latex?%5Cmu_%7Bij%7D%20(b_i)">가 비선형이기 때문에 적분이 <strong>closed-form</strong>으로 풀리지 않는 경우가 대부분이라는 것이고, 따라서 실제로는 이 적분을 <strong>수치적(또는 근사적)</strong>으로 계산한 뒤, 그 결과(근사치)를 최대화하여 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cbeta%7D,%20%5Chat%7B%5Cmathbf%7BG%7D%7D,%20%5Chat%7B%5Cphi%7D">를 구하게 됩니다.</p>
<section id="glmms-parameter-추정marginal-likelihood-approximation" class="level4"><h4 class="anchored" data-anchor-id="glmms-parameter-추정marginal-likelihood-approximation">GLMM’s parameter 추정(Marginal Likelihood &amp; Approximation)</h4>
<hr>
<p>다시 한 번 언급하자면 문제는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i">가 랜덤효과이므로 이를 적분해야 한다는 것인데, <img src="https://latex.codecogs.com/png.latex?%5Cmu_%7Bij%7D(%5Cmathbf%7Bb%7D_i)">가 <strong>비선형</strong>이기 때문에 이 적분이 <strong>closed-form</strong>으로 표현되지 않는 것이고, 다음과 같은 <strong>근사화</strong> 기법을 사용하여 계산합니다.</p>
<ul>
<li><p><strong>Laplace Approximation</strong><br><img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7Bb%7D_i"> 주변에서 2차 근사를 수행하여 적분을 근사화하는 방법입니다. 한 번(1차) 또는 고차(AGQ, Adaptive Gauss-Hermite Quadrature) 버전으로 더 정확하게 시도할 수 있습니다.</p></li>
<li><p><strong>Gauss-Hermite Quadrature</strong><br>
적분을 수치적(Numerical)으로 가까운 근사값으로 계산합니다. 무작위 효과 차원이 높아질수록 계산량이 기하급수적으로 늘어날 수 있으므로, 실무에서는 차원이 작은 랜덤 효과 구조(예: 랜덤 인터셉트만)에서 자주 사용합니다.</p></li>
<li><p><strong>Penalized Quasi-Likelihood (PQL)</strong><br>
고전적으로 제안된 근사 기법으로, GLM의 IRLS 절차를 변형하여 무작위효과를 순차적으로 추정합니다. 데이터가 크거나, 근사 정밀도가 크게 중요하지 않은 상황에서 가볍게 쓰일 수 있습니다.</p></li>
</ul>
<p>최종적으로, (1) 적분으로 정의된 <strong>marginal likelihood</strong>를 (2) 수치적 근사화를 통해 (3) 최적화(예: Newton-Raphson, EM 등)하여, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7BG%7D%7D">, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cphi%7D"> 등을 찾습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D,%20%5C;%5Chat%7B%5Cboldsymbol%7BG%7D%7D,%20%5C;%5Chat%7B%5Cphi%7D%0A=%20%5Cunderset%7B%5Cboldsymbol%7B%5Cbeta%7D,%20%5Cboldsymbol%7BG%7D,%20%5Cphi%7D%7B%5Cmathrm%7Bargmax%7D%7D%0A%5C;%5C;%5CBigl%5C%7B%0A%5Cprod_%7Bi=1%7D%5E%7BK%7D%0A%5Cint%0A%5Cprod_%7Bj=1%7D%5E%7Bn_i%7D%20f%5Cbigl(Y_%7Bij%7D%5Cmid%20%5Cmu_%7Bij%7D(%5Cmathbf%7Bb%7D_i),%20%5Cphi%5Cbigr)%0A%5C,%20%5Cvarphi(%5Cmathbf%7Bb%7D_i)%5C,%20d%5Cmathbf%7Bb%7D_i%0A%5CBigr%5C%7D.%0A"></p>
</section><section id="glmm-vs.-gee" class="level4"><h4 class="anchored" data-anchor-id="glmm-vs.-gee">GLMM vs.&nbsp;GEE</h4>
<hr>
<p>이 data간 상관관계를 고려하기 위해 개발된 두 모델을 짧게 정리해보면, <strong>GEE</strong>는 “Population-Average” 접근으로 군집 내 상관을 <strong>working correlation</strong> 방식으로 모델링하며, 완전한 joint PDF를 명시하지 않고 Quasi-likelihood처럼 추정하는 기법이었고, GLMM은 “Subject-Specific” 접근으로 군집/클러스터 효과를 <strong>무작위 효과</strong>로 모델링하여 종속변수를 (조건부) Exponential Family distribution으로 가정하고, 이 likelihood를 marginal하게 적분함으로써 추정합니다.<br></p>
</section></section><section id="glmm-parameters-variance" class="level3"><h3 class="anchored" data-anchor-id="glmm-parameters-variance">3.4. GLMM parameter’s Variance</h3>
<hr>
<p>마지막으로, GLMM에서의 추정된 파라미터(고정 효과 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">, 무작위 효과 분산-공분산 행렬 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7BG%7D"> )의 분산 추정(표준 오차, 신뢰구간 등) 방법을 보겠습니다. GLMM의 경우, 근사화하여 최대화한 <strong>marginal log-likelihood</strong>에서의 <strong>헤시안 행렬(Hessian)</strong>을 기반으로 고정효과 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D"> 의 분산을 추정할 수 있습니다. 구체적으로, 아래와 같은 일반적 형식을 취합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Coperatorname%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%0A=%0A%5Cbigl%5B%0A%20%20-%5Cnabla%5E2_%7B%5Cboldsymbol%7B%5Cbeta%7D,%5Cboldsymbol%7B%5Cbeta%7D%7D%0A%20%20%20%5C,%5Cell(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D,%20%5Chat%7B%5Cboldsymbol%7BG%7D%7D,%20%5Chat%7B%5Cphi%7D)%0A%5Cbigr%5D%5E%7B-1%7D,%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Cell">은 GLMM의 (근사) marginal log-likelihood, <img src="https://latex.codecogs.com/png.latex?%5Cnabla%5E2_%7B%5Cboldsymbol%7B%5Cbeta%7D,%5Cboldsymbol%7B%5Cbeta%7D%7D">는 고정효과 파라미터 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">에 대한 2차 미분(Hessian)으로, 이 헤시안을 (적절한 수치 방법으로) 근사화하여 얻고, 그 역행렬이 분산 추정의 결과에 해당합니다. <strong>이는 여전히 log likelihood을 통한 추정이기 때문에 Fisher information matrix로 분산을 추정</strong>한다고 생각하면 될 것 같습니다. GEE(2장에서)와 마찬가지로, GLMM에서도 모델의 설계에서의 작은 misspecification이 있을 가능성을 고려하여 안정적으로 Sandwich estimator를 통해 추정할 수 있는지 고민할 수 있습니다. GLMM의 경우, 군집 간 독립 이나 군집 내 random effect의 정규성 가정과 같은 가정이 크게 벗어나지 않는다고 믿으면 위 <strong>모델 기반(model-based)</strong> 추정 분산을 사용하면 되고, 그렇지 않은 <em>“무작위 효과 분포가 정규가 아닐 가능성”</em> 혹은 <em>“link/variance function 형태가 부정확할 가능성”</em> 등을 고려하기 위해 적절한 <strong>샌드위치 추정(sandwich-type variance)</strong> 기법을 시도할 수도 있습니다. 다만, GEE와 달리 GLMM에서의 robust variance estimation은 쉽게 구현되지 않으며, <strong>근사기법, 부트스트랩(bootstrap)</strong> 등을 통해 대안적으로 접근하는 사례도 많습니다.</p>
<p>random effect의 분산-공분산 행렬 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7BG%7D"> 또한 우도(또는 제한 우도)에서 <strong>편미분이 0</strong> 조건을 이용하여 추정하지만, 그 표준 오차(불확실성)를 추정하는 과정 역시 (1) 2차 미분, (2) 프로파일(profile) likelihood, (3) 수치적 근사화 등을 거쳐야 합니다.<br></p>
</section></section><section id="r-코드-예제-gee-glmm" class="level2"><h2 class="anchored" data-anchor-id="r-코드-예제-gee-glmm">4. R 코드 예제: GEE, GLMM</h2>
<p>아래 R 코드를 복사하여 로컬 환경에서 돌려보세요.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(nlme)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#data(Orthodont)  # 치아 성장 데이터 (클러스터: Subject)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#Orthodont$binary &lt;- ifelse(Orthodont$distance &gt; 25, 1, 0)  # 이항 변환</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## GEE 모델 적합 (Exchangeable 상관 구조)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(geepack)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#gee_fit &lt;- geeglm(binary ~ age + Sex,</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#                 id = Subject,          # 클러스터 변수</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#                 data = Orthodont,</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#                 family = binomial,</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#                 corstr = "exchangeable")  # 동일 상관 가정</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#summary(gee_fit)  # 결과 출력</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## GLMM 모델 적합 (랜덤 절편 모델)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(lme4)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#glmm_fit &lt;- glmer(binary ~ age + Sex + (1|Subject),  # 랜덤 절편</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#                 data = Orthodont,</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#                 family = binomial)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#summary(glmm_fit)  # 결과 출력</span></span></code></pre></div></div>
</div>
</section><section id="마무리하며" class="level2"><h2 class="anchored" data-anchor-id="마무리하며">마무리하며</h2>
<p>이번 장에서는 M-estimation 개념부터 시작하여, GLM이 어떻게 “estimating equation”의 한 사례로 해석되는지, GEE가 GLM을 확장하여 상관구조를 모델링하고, robust 분산을 제공함으로써 군집/반복측정 데이터를 다루는 과정을, GLMM이 임의효과를 통해 계층적 구조를 명시적으로 모델링하는 방식을 자세히 살펴보았습니다. 그리고 샌드위치 추정량(robust variance) 형태가 M-estimation의 일반 이론에서 비롯된다는 점도 수식과 함께 설명했습니다.</p>
<p>정리하자면, M-estimation은 MLE, OLS, GEE, GLMM 모두를 포괄하는 추정 이론적 틀로서, 샌드위치 분산은 그 점근 정규성(Asymptotic Normality)의 결과물이며, GEE는 marginal mean에 주목하고 robust한 표준오차를 산출해주는 반면, GLMM은 임의효과를 통해 개체별(군집별) 차이를 직접 모델링합니다. 실제 데이터 분석에서는 연구 목적(개체별 효과 추정 vs 전체 평균 효과 추정), 데이터 특성(정확한 상관 구조 가정 vs 모형 가정의 유연성) 등을 종합하여 GEE와 GLMM 중 적절한 접근을 택하거나 비교하는 것이 중요합니다. 사실 Regression Model에는 이번 블로그 “Exploring Regression Models for Regression Analysis”에서 다룬 모델들을 제외하고도 아주 다양한 철학과 수식을 가진 모델들이 있습니다. 다만 여기서는 의학 분석에서 자주 사용되는 모델을 다루었으며, 이를 어느 정도 이해하셨다면 이외의 모델을 이해하는 데에 부족함이 없을 것이라고 생각합니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{seungjun2025,
  author = {Seungjun, Lee},
  title = {Exploring {Regression} {Models} for {Regression} {Analysis}
    (3): {GEE,} {GLMM,} {M-statistics,} {Robust} (Sandwich) Estimation},
  date = {2025-02-28},
  url = {https://blog.zarathu.com/posts/2025-02-28-reg3/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-seungjun2025" class="csl-entry quarto-appendix-citeas">
Seungjun, Lee. 2025. <span>“Exploring Regression Models for Regression
Analysis (3): GEE, GLMM, M-Statistics, Robust (Sandwich)
Estimation.”</span> February 28, 2025. <a href="https://blog.zarathu.com/posts/2025-02-28-reg3/">https://blog.zarathu.com/posts/2025-02-28-reg3/</a>.
</div></div></section></div> ]]></description>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-02-28-reg3/</guid>
  <pubDate>Fri, 28 Feb 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-02-28-reg3/img/reg3.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Exploring Regression Models for Regression Analysis (2): GLM, Exponential Family, Link Function, IRLS(Fisher scoring), Cluster-robust standard error</title>
  <dc:creator>Lee Seungjun</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-02-28-reg2/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="들어가며" class="level2"><h2 class="anchored" data-anchor-id="들어가며">들어가며</h2>
<p>2장에서는 1장에서 다룬 기본 linear regression에서 link function을 도입하여 regression의 개념을 outcome of single yes/no, outcome of single K-way, count 등 non-normal한 종속변수로 확장한 Generalized linear model의 개념을 Exponential Family, Link Function와 같은 핵심개념과 함께 깊게 살펴보며, model의 parameter를 estimate하는 알고리즘인 IRLS(Fisher scoring) 집중적으로 estimation methods를 소개합니다. 이후, HC standard errors의 clustered data 버전인 Cluster-robust standard error를 보고 마치겠습니다.<br></p>
</section><section id="generalized-linear-models-glms" class="level2"><h2 class="anchored" data-anchor-id="generalized-linear-models-glms">1. Generalized Linear Models (GLMs)</h2>
<section id="linear-model-한계" class="level3"><h3 class="anchored" data-anchor-id="linear-model-한계">1.1. Linear Model 한계</h3>
<hr>
<p>1장에서 본 <strong>Linear Regression Model</strong>은 (1) 선형성(Linearity) 가정, (2) 오차항의 정규성(Normality) 가정, (3) 오차항의 독립성(Independence) 가정, (4) 오차항의 등분산성(Homoscedasticity) 가정에서 비롯된 모델이었고, Heteroskedasticity-Consistent Standard Errors (HC Standard Errors)를 통해 오차항의 등분산성(Homoscedasticity) 가정이 깨진 data에 대해서도 Linear model로부터 얻은 모델 parameter의 분산을 robust하게 추정할 수 있었습니다. 그러나, 위에서 언급하였듯 <strong>outcome of single yes/no, outcome of single K-way, count 등 많은 data는 반응 변수 Y가 정규분포를 따르지 않거나 등분산성 가정, 선형성에 위배</strong>됩니다. 각각에 대해서 좀 더 설명하자면, 어떤 사건이나 행동이 일어나거나 그렇지 않은 경우를 고려하는 <strong>이진 데이터(binary data, outcome of single yes/no)</strong>의 경우, <img src="https://latex.codecogs.com/png.latex?Y%20%5Cin%20%5C%7B0,%201%5C%7D">로 제한되며 이를 <img src="https://latex.codecogs.com/png.latex?Y%20%5Cin%20%5Cmathbb%7BR%7D">인 정규분포로 가정하는 것은 옳지 않습니다. 특정 기간 동안 발생하는 사건의 횟수 등, 이진 분류처럼 discrete한 종속변수 값을 가지는 <strong>카운트 데이터(count data)</strong> 또한 discrete(정수) 값만 갖으며, 이 두 경우는 종종 분산이 평균(모델의 예측)에 비례하는 형태를 갖을 수 있고, 이는 당연하게도 등분산성 가정을 위배합니다.</p>
<p>이러한 데이터의 경우 단순히 독립변수의 선형결합 형태, 또는 기하학적으로는 Hyper plane 형태로 모델을 만들면, 비선형적인 (이진 데이터 등) 위 같은 경우에 대해서는 올바르게 고려하지 못할 것입니다. 이러한 기존의 Linear Regression 모델의 한계를 극복하고, (종속변수의) 다양한 형태의 데이터를 모델링하기 위해 여러 함수를 설계함으로써 유연성을 확장한 <strong>Generalized Linear Models</strong>이 개발되었습니다. <strong>Generalized Linear Models(GLMs)</strong>의 중요 구성 요소들과 원리를 간략히 설명해보자면, 선형 결합으로 바로 종속변수를 예측하는 대신, non-linear한 <strong>Link Function</strong>에 넣어 최종적으로 예측함으로써 non-linear한 종속변수에도 fit 할 수 있고, 이에 따라 종속변수의 분포가 <strong>정규분포가 아닌 다른 분포(Exponential Family)</strong>도 포함할 수 있도록 하였으며, 이 <strong>Exponential Family와 Variance function</strong>구성은 종속변수의 분산이 모델의 예측값(종속변수의 mean)마다 다를 수 있도록 합니다. 이를 통해 <strong>Generalized Linear Models</strong>는 위 네 개의 Linear Regression 가정 중 (1) 선형성(Linearity) 가정, (2) 오차항의 정규성(Normality) 가정, (4) 오차항의 등분산성(Homoscedasticity) 가정을 깼으며, 위에서 Linear Model의 한계로 언급한 데이터들을 고려할 수 있는 모델입니다.</p>
</section><section id="glm-정의-및-수학적-표현" class="level3"><h3 class="anchored" data-anchor-id="glm-정의-및-수학적-표현">1.2. GLM 정의 및 수학적 표현</h3>
<hr>
<p><strong>GLM</strong>은 세 가지 구성 요소 (Random component, Systematic component, Link function)으로 정의 되며, 이때 Random component는 Y를 Exponential Family로, Systematic component는 Linear predictor와 Link function으로 구성됩니다. 어떻게 <strong>Generalized Linear Models</strong>가 설계되었는지 component들을 하나하나 자세히 다뤄보겠습니다.</p>
<section id="linear-predictor" class="level4"><h4 class="anchored" data-anchor-id="linear-predictor">Linear predictor</h4>
<hr>
<p>Linear predictor <img src="https://latex.codecogs.com/png.latex?%5Ceta">는 말그대로 Linear Model처럼 <strong>모델 parameter</strong> <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D"><strong>와 독립변수</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D"><strong>의 선형 결합</strong>으로, 기존에는 <img src="https://latex.codecogs.com/png.latex?%5Ceta">로 바로 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7By%7D">를 추정하여 non-linear한 종속변수를 고려하지 못하였었다면, GLM은 <img src="https://latex.codecogs.com/png.latex?%5Ceta">를 계산한 후, 이 값을 <em>non-linear한 Link function에 input하여 최종적으로 종속변수를 예측</em>합니다. 중요한 점은, 이는 단순히 종속변수를 transform한 뒤(로그 등) 이전처럼 선형적으로 추정하는 Transformation (with LM)과 다르다는 점입니다. 가장 큰 차이점은 Transformation을 함으로써 종속변수의 sample space에서 boundaries에 있는 값들은 정의가 되지 않고(로그는 0에서 정의되지 않음.), 이후 바로 Linear Model을 사용하기 위해선 종속변수가 변형 이후 반드시 linearity와 variance의 homogeneity가 거의 보장되어야 합니다.(기존 LM을 사용하기 때문에 이때 사용한 가정 또한 필요하게 됩니다.) GLM의 Linear predictor (선형 예측자) <img src="https://latex.codecogs.com/png.latex?%5Ceta">의 식은 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%5Ceta_i%20=%20%5Cbeta_0%20+%20%5Cbeta_1%20x_%7B1i%7D%20+%20%5Cbeta_2%20x_%7B2i%7D%20+%20%5Cdots%20+%20%5Cbeta_p%20x_%7Bpi%7D"></p>
</section><section id="link-function-링크-함수" class="level4"><h4 class="anchored" data-anchor-id="link-function-링크-함수">Link function (링크 함수)</h4>
<hr>
<p>Link function <img src="https://latex.codecogs.com/png.latex?g(%5Cmu_i)">는 <strong>non-linear하고 미분 및 inverse(역)가 가능한 함수로 정의</strong>되며, 종속변수의 평균 <img src="https://latex.codecogs.com/png.latex?E(Y_i)%20=%20%5Cmu_i">를 선형 예측자 <img src="https://latex.codecogs.com/png.latex?%5Ceta_i">와 연결하여 간접적으로 <strong>독립변수 및 모델 parameter의 선형결합과 종속변수를 mapping하는 역할</strong>을 합니다.<br><img src="https://latex.codecogs.com/png.latex?g(%5Cmu_i)%20=%20%5Ceta_i"></p>
</section><section id="variance-function-분산-함수" class="level4"><h4 class="anchored" data-anchor-id="variance-function-분산-함수">Variance function (분산 함수)</h4>
<hr>
<p>분산 함수는 <strong>평균</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmu_i"><strong>에 따라 종속변수의 분산이 어떻게 변하는지</strong>를 나타냅니다. 이를 통해 간접적으로 독립변수에 따라 분산이 다르게 나오는 것을 반영할 수 있으며 식은 아래와 같고,<br><img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7BVar%7D(Y_i)%20=%20%5Cphi%20V(%5Cmu_i)"><br></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Cphi">는 <strong>dispersion parameter</strong>로, 일반적으로 특정 분포에 따라 다르게 정의됩니다. (예: Poisson 분포에서는 <img src="https://latex.codecogs.com/png.latex?%5Cphi%20=%201">).</p>
</section><section id="exponential-family" class="level4"><h4 class="anchored" data-anchor-id="exponential-family">Exponential Family</h4>
<hr>
<p>GLM은 종속변수의 분포로 Gaussian(또는 정규분포)를 포함한, 더욱 general한 <strong>Exponential Family</strong>을 고려하며, 이 분포는 다음과 같은 일반 형태를 가집니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Af(y;%20%5Ctheta,%20%5Cphi)%20=%20%5Cexp%20%5Cleft%5C%7B%20%5Cfrac%7By%20%5Ctheta%20-%20b(%5Ctheta)%7D%7B%5Cphi%7D%20+%20c(y,%20%5Cphi)%20%5Cright%5C%7D%0A"></p>
<p>즉 GLM은 LM과 다르게 linear predictor, link function, variance function을 설계함으로써 종속변수가 더욱 general한 분포인 exponential family distribution인 경우에도 잘 mapping할 수 있도록 하는 모델이라고 볼 수 있습니다. 위 식에서 의미론적으로 각 parameters를 해석하면 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">는 <strong>canonical parameter</strong>로 분포의 위치를 나타내는 파라미터, <img src="https://latex.codecogs.com/png.latex?%5Cphi">는 dispersion parameter로 분산과 관련된 파라미터, <img src="https://latex.codecogs.com/png.latex?b(%5Ctheta)">는 평균과 분산 관계를 정의하는 함수입니다. 이 분포에 대해 <img src="https://latex.codecogs.com/png.latex?E(Y)%20=%20b'(%5Ctheta)%20=%20%5Cmu">, <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7Bvar%7D(Y)%20=%20%5Cphi%20b''(%5Ctheta)%20=%20%5Cphi%20V(%5Cmu)">이라는 특성이 증명 가능하고, 이는 “2. GLMs 추정”에서 모델 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">를 추정하는 과정에 필요하기 때문에 아래에서 증명할 것입니다. 이보다 더욱 general한 분포로 (dispersion parameter 관련) exponential dispersion family가 있습니다.</p>
<p>다음으로 넘어가기 전에 간단하게 잘 알려져있는 Exponential Family의 예시인 정규분포, 이항분포, 포아송분포, 감마분포가 이에 포함됨을 확인해보겠습니다.</p>
<p><strong>(1) 정규분포 (Normal Distribution)</strong></p>
<p>정규분포의 확률밀도함수는 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20%5Cmu,%20%5Csigma%5E2)%20=%20%5Cfrac%7B1%7D%7B%5Csqrt%7B2%5Cpi%5Csigma%5E2%7D%7D%20%5Cexp%5Cleft%5C%7B%20-%5Cfrac%7B(y%20-%20%5Cmu)%5E2%7D%7B2%5Csigma%5E2%7D%20%5Cright%5C%7D.%0A%20%20%20%20"> 이를 Exponential Family 형태로 변환하면: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20%5Cmu,%20%5Csigma%5E2)%20=%20%5Cexp%5Cleft%5C%7B%20%5Cfrac%7By%5Cmu%20-%20%5Cfrac%7B%5Cmu%5E2%7D%7B2%7D%7D%7B%5Csigma%5E2%7D%20-%20%5Cfrac%7By%5E2%7D%7B2%5Csigma%5E2%7D%20-%20%5Cfrac%7B1%7D%7B2%7D%20%5Clog(2%5Cpi%5Csigma%5E2)%20%5Cright%5C%7D.%0A%20%20%20%20"></p>
<ul>
<li>Canonical parameter: <img src="https://latex.codecogs.com/png.latex?%5Ctheta%20=%20%5Cmu">,</li>
<li>Dispersion parameter: <img src="https://latex.codecogs.com/png.latex?%5Cphi%20=%20%5Csigma%5E2">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b(%5Ctheta)%20=%20%5Cfrac%7B%5Ctheta%5E2%7D%7B2%7D">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b'(%5Ctheta)%20=%20%5Ctheta%20=%20%5Cmu">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b''(%5Ctheta)%20=%201">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?c(y,%20%5Cphi)%20=%20-%5Cfrac%7By%5E2%7D%7B2%5Cphi%7D%20-%20%5Cfrac%7B1%7D%7B2%7D%20%5Clog(2%5Cpi%5Cphi)">.</li>
</ul>
<p><strong>(2) 이항분포 (Binomial Distribution)</strong></p>
<p>이항분포의 확률질량함수는 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20n,%20p)%20=%20%5Cbinom%7Bn%7D%7By%7D%20p%5Ey%20(1-p)%5E%7Bn-y%7D.%0A%20%20%20%20"> 이를 Exponential Family 형태로 변환하면: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20n,%20p)%20=%20%5Cexp%5Cleft%5C%7B%20y%20%5Clog%5Cleft(%5Cfrac%7Bp%7D%7B1-p%7D%5Cright)%20+%20n%20%5Clog(1-p)%20+%20%5Clog%5Cbinom%7Bn%7D%7By%7D%20%5Cright%5C%7D.%0A%20%20%20%20"></p>
<ul>
<li>Canonical parameter: <img src="https://latex.codecogs.com/png.latex?%5Ctheta%20=%20%5Clog%5Cleft(%5Cfrac%7Bp%7D%7B1-p%7D%5Cright)">,</li>
<li>Dispersion parameter: <img src="https://latex.codecogs.com/png.latex?%5Cphi%20=%201">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b(%5Ctheta)%20=%20n%20%5Clog(1%20+%20e%5E%5Ctheta)">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b'(%5Ctheta)%20=%20e%5E%5Ctheta%20=%20%5Clambda">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b''(%5Ctheta)%20=%20e%5E%5Ctheta%20=%20%5Clambda">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?c(y,%20%5Cphi)%20=%20%5Clog%5Cbinom%7Bn%7D%7By%7D">.</li>
</ul>
<p><strong>(3) 포아송분포 (Poisson Distribution)</strong></p>
<p>포아송분포의 확률질량함수는 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20%5Clambda)%20=%20%5Cfrac%7B%5Clambda%5Ey%20e%5E%7B-%5Clambda%7D%7D%7By!%7D.%0A%20%20%20%20"> 이를 Exponential Family 형태로 변환하면: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20%5Clambda)%20=%20%5Cexp%5Cleft%5C%7B%20y%20%5Clog%5Clambda%20-%20%5Clambda%20-%20%5Clog(y!)%20%5Cright%5C%7D.%0A%20%20%20%20"></p>
<ul>
<li>Canonical parameter: <img src="https://latex.codecogs.com/png.latex?%5Ctheta%20=%20%5Clog%5Clambda">,</li>
<li>Dispersion parameter: <img src="https://latex.codecogs.com/png.latex?%5Cphi%20=%201">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b(%5Ctheta)%20=%20e%5E%5Ctheta">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b'(%5Ctheta)%20=%20%5Cfrac%7Bn%20e%5E%5Ctheta%7D%7B1%20+%20e%5E%5Ctheta%7D%20=%20np">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b''(%5Ctheta)%20=%20%5Cfrac%7Bn%20e%5E%5Ctheta%7D%7B(1%20+%20e%5E%5Ctheta)%5E2%7D%20=%20np(1-p)">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?c(y,%20%5Cphi)%20=%20-%5Clog(y!)">.</li>
</ul>
<p><strong>(4) 감마분포 (Gamma Distribution)</strong></p>
<p>감마분포의 확률밀도함수는 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20k,%20%5Ctheta)%20=%20%5Cfrac%7B1%7D%7B%5CGamma(k)%5Ctheta%5Ek%7D%20y%5E%7Bk-1%7D%20e%5E%7B-%5Cfrac%7By%7D%7B%5Ctheta%7D%7D.%0A%20%20%20%20"> 이를 Exponential Family 형태로 변환하면: <img src="https://latex.codecogs.com/png.latex?%0A%20%20%20%20f(y;%20k,%20%5Ctheta)%20=%20%5Cexp%5Cleft%5C%7B%20-%5Cfrac%7By%7D%7B%5Ctheta%7D%20+%20(k-1)%5Clog%20y%20-%20k%5Clog%5Ctheta%20-%20%5Clog%5CGamma(k)%20%5Cright%5C%7D.%0A%20%20%20%20"></p>
<ul>
<li>Canonical parameter: <img src="https://latex.codecogs.com/png.latex?%5Ctheta%20=%20-%5Cfrac%7B1%7D%7B%5Ctheta%7D">,</li>
<li>Dispersion parameter: <img src="https://latex.codecogs.com/png.latex?%5Cphi%20=%20%5Cfrac%7B1%7D%7Bk%7D">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b(%5Ctheta)%20=%20-%5Clog(-%5Ctheta)">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b'(%5Ctheta)%20=%20%5Cfrac%7B1%7D%7B%5Ctheta%7D%20=%20%5Cmu">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?b''(%5Ctheta)%20=%20%5Cfrac%7B1%7D%7B%5Ctheta%5E2%7D%20=%20%5Cmu%5E2">,</li>
<li>
<img src="https://latex.codecogs.com/png.latex?c(y,%20%5Cphi)%20=%20(k-1)%5Clog%20y%20-%20%5Clog%5CGamma(k)">.</li>
</ul>
<p>위에서 <img src="https://latex.codecogs.com/png.latex?b(%5Ctheta)">는 한 번 미분하면 mean, 두 번 미분하면 variance의 term과 관련됨을 언급하였고 위 4개의 분포에서 원래 알고 계신 mean, variance와 <img src="https://latex.codecogs.com/png.latex?b'(%5Ctheta)">, <img src="https://latex.codecogs.com/png.latex?b''(%5Ctheta)">가 dispersion parameter를 고려하면 일치한 것을 확인하실 수 있습니다. 이는 cumulant generating function의 일부이기 때문이며, 따라서 평균과 분산 관계를 정의하는 항이라고 언급하였던 것입니다.</p>
</section><section id="canonical-link" class="level4"><h4 class="anchored" data-anchor-id="canonical-link">Canonical Link</h4>
<hr>
<p><strong>Canonical link</strong>는 GLM에서 통계적 성질을 최적화하기 위해 사용되는 링크 함수(link function)로, 다음과 같이 정의됩니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Ag(%5Cmu_i)%20=%20g(b'(%5Ctheta_i))%20=%20%5Ctheta_i%20=%20%5Ceta_i%0A"> 이 식의 의미는 결국 아래 식과 같습니다. <img src="https://latex.codecogs.com/png.latex?%0Ag%20=%20(b')%5E%7B-1%7D%0A"></p>
<p>아래에서 확인하겠지만, <strong>Binomial 분포</strong>의 경우 canonical link는 <strong>logit</strong> 함수이고, <strong>Poisson 분포</strong>의 경우 canonical link는 <strong>log</strong> 함수이며, Canonical link를 사용하면 MLE(Maximum Likelihood Estimation) 과정이 단순화되고, efficient한 추정치를 얻을 수 있기 때문에 link function은 거의 항상 Canonical link로 정의합니다. 또한, canonical하지 않은 link function의 경우에도 위 Exponential Family distribution에서 식 조작을 통해 canonical link 형태를 만들 수 있습니다.</p>
</section></section><section id="glm-예시" class="level3"><h3 class="anchored" data-anchor-id="glm-예시">1.3. GLM 예시</h3>
<hr>
<p>위 철학에 따라, data가 따르는 Exponential Family 중 특정 분포가 정해지면, 이에 해당하는 보통 사용하는 Link function(Canonical link), Variance function가 정해져 있고 결국 모델이 특정되며, GLM은 이렇게 특정될 수 있는 모든 모델에서 공통적으로 parameter와 그 variance를 추정해내는 general한 모델이라고 생각할 수 있습니다. 여기에서 다루지는 않겠지만, 사실 특정한 형태의 data에서 가능한 link function은 여러 개이며, 이에 따라 variance function도 여러 가지가 가능할 수 있습니다. 그러나 효율성과 computation cost를 고려하여 보통 사용되는 function forms는 정해져 있다고 알아두시면 좋을 것 같습니다. 아래 예시 중 대표적으로 Binomial 예시에서는 link function이 0 이상 1 이하의 정의역에서 실수 전체(for linear predictor)를 map할 수 있는 미분 및 역이 가능한 함수이면 되지만, 보통 logit function이 사용됩니다. 헷갈릴 수 있지만 아래 Exponential Family 중 친숙한 분포의 예시를 직관적인 관점에서 고려하여 위에서 얻은 Exponential Family의 form과 같은 결과가 나옴을 보시면 좋을 것 같습니다.<br></p>
<p>아래의 예시에서 linear predictor는 공통이므로 미리 정의하고 각각의 link function, variance function은 어떻게 특정되는지를 보겠습니다.<br></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ceta_i%20=%20%5Cbeta_0%20+%20%5Cbeta_1%20x_%7B1i%7D%20+%20%5Cdots%20+%20%5Cbeta_p%20x_%7Bpi%7D%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Ceta_i">는 linear predictor, <img src="https://latex.codecogs.com/png.latex?%5Cbeta_0,%20%5Cbeta_1,%20...,%20%5Cbeta_p">는 regression coefficients(parameters), <img src="https://latex.codecogs.com/png.latex?x_%7B1i%7D,%20...,%20x_%7Bpi%7D">는 predictor variables(독립변수) 입니다.</p>
<section id="binomial-case" class="level4"><h4 class="anchored" data-anchor-id="binomial-case">Binomial Case</h4>
<hr>
<p>Binomial Data, 즉 종속변수가 <img src="https://latex.codecogs.com/png.latex?Y_i%20%5Csim%20%5Ctext%7BBinomial%7D(n_i,%20p_i)">인 data의 경우, 종속변수의 기댓값의 sample space 또한 0~1이며, 우리가 모델링하고 싶은 값이 <img src="https://latex.codecogs.com/png.latex?Y_i/n_i">인 경우를 상정해보겠습니다. 직관적으로 의미로부터 functions가 어떻게 되어야 할 지 생각해보면, <img src="https://latex.codecogs.com/png.latex?E(Y_i%20/%20n_i)%20=%20p_i">이고, 분산은 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B1%7D%7Bn_i%7D%20p_i%20(1%20-p_i)"> 입니다. <img src="https://latex.codecogs.com/png.latex?Y_i/n_i">의 variance 식에 <img src="https://latex.codecogs.com/png.latex?Y_i/n_i">의 mean이 들어감을 알 수 있고, 기존 LM에서는 이렇게 관측치에 따라 다르게 variance를 고려할 수 없었지만, GLM에서는 이 관계를 variance function을 통해 고려할 수 있으며, 식은 다음과 같습니다: <img src="https://latex.codecogs.com/png.latex?%0AV(%5Cmu_i)%20=%20%5Cmu_i%20(1%20-%20%5Cmu_i)%0A"> 또한, <img src="https://latex.codecogs.com/png.latex?Y_i%20/%20n_i">와 linear predictor를 matching 해줄 수 있는 미분가능한 function을 link로 고려해야 하고, Binomial에서는 non-linear link function으로 대부분 <strong>logit 함수</strong>를 사용합니다. (이때, logit function의 inverse는 sigmoid function입니다.)</p>
<p><img src="https://latex.codecogs.com/png.latex?g(%5Cmu_i)%20=%20%5Clog%20%5Cleft(%20%5Cfrac%7B%5Cmu_i%7D%7B1%20-%20%5Cmu_i%7D%20%5Cright)"></p>
<p>이 식은 위에서 확인한 이항분포의 canonical parameter와 같은 형태임을 알 수 있습니다.</p>
</section><section id="poisson-case" class="level4"><h4 class="anchored" data-anchor-id="poisson-case">Poisson Case</h4>
<hr>
<p>Poisson Data, 즉 종속변수가 <img src="https://latex.codecogs.com/png.latex?Y_i%20%5Csim%20%5Ctext%7BPoisson%7D(%5Clambda_i)">인 data이고 우리가 모델링하고 싶은 값이 종속변수 <img src="https://latex.codecogs.com/png.latex?Y_i">인 경우를 상정해보겠습니다. 직관적으로 의미로부터 functions가 어떻게 되어야 할 지 생각해보면, <img src="https://latex.codecogs.com/png.latex?E(Y_i)%20=%20%5Clambda_i">이고 분산은 <img src="https://latex.codecogs.com/png.latex?%5Clambda_i"> 이므로, 마찬가지로 <img src="https://latex.codecogs.com/png.latex?Y_i">의 variance 식에 <img src="https://latex.codecogs.com/png.latex?Y_i">의 mean이 들어감을 알 수 있고, variance function은 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AV(%5Cmu_i)%20=%20%5Cmu_i%0A"> 또한, <img src="https://latex.codecogs.com/png.latex?Y_i">의 sample space는 0 이상의 실수로, 이와 linear predictor를 matching 해줄 수 있는 미분가능한 non-linear function을 link로 고려해야 하고, Binomial에서는 link function으로 대부분 <strong>log 함수</strong>를 사용합니다. (inverse는 지수 함수.) <img src="https://latex.codecogs.com/png.latex?%0Ag(%5Cmu_i)%20=%20%5Clog(%5Cmu_i)%0A">이 식은 위에서 확인한 포아송분포의 canonical parameter와 같은 형태임을 알 수 있습니다. 위 두 예시에서는 어떻게 GLMs의 구성 요소들이 선택되는지 직관적으로 보았고, 이는 이해를 돕기 위한 해석이었으며 이미 위에서 Exponential Family에 포함됨을 보일 때 같은 결과가 나왔다는 것을 보시면 됩니다. 위 Binomial Data의 GLM은 Logistic Regression, Poisson Data의 GLM은 Poisson Regression으로도 불립니다.<br></p>
</section></section></section><section id="glms-추정" class="level2"><h2 class="anchored" data-anchor-id="glms-추정">2. GLMs 추정</h2>
<p>위 내용들을 통해서 GLMs가 어떻게 비정규분포를 갖는 종속변수를 고려해서 잘 작동하며, 어떠한 함수(Link function, Variance function, Exponential Family, Canonical Link)가 어떠한 수식과 철학으로 GLM을 구성하고 있는지 확인할 수 있었습니다. GLMs은 거의 대부분의 고려 가능한 data 분포가 Exponential Family를 따르며, 이에 대해 일관적인 form과 parameter estimation이 가능하기 때문에 아주 powerful한 Regression Model입니다. 그러나 어떻게 Exponential Family를 따르는 data를 다룰 수 있는지는 확인할 수 있었지만, <em>어떻게 Regression Model의 parameter와 그 분산을 추정할 수 있는지는 다루지 않았습니다.</em> <strong>Linear Model에서는 closed-form solution을 쉽게 찾을 수 있었지만, GLM은 대부분의 경우(있는 경우도 있습니다.) 이러한 closed-form이 없어 컴퓨터 프로그램으로 여러 번에 걸쳐 추정할 수 있도록 알고리즘을 구현하여 이를 추정</strong>합니다. 실제로 이 estimation의 수식과 실제 구현 과정을 다루기 위해서는 긴 증명 과정을 거치는데, 최대한 중요한 부분은 빠지지 않으면서 증명해보겠습니다. 우선, <strong>MLE로 모델을 추정하는 과정을 증명하기 위해 필요한 두 가지 유용한 성질</strong>을 살펴보겠습니다. (Derivatives of Log Likelihood’s, Exponential Family 성질)<br></p>
<section id="derivatives-of-log-likelihoods-성질" class="level3"><h3 class="anchored" data-anchor-id="derivatives-of-log-likelihoods-성질">2.1. Derivatives of Log Likelihood’s 성질</h3>
<hr>
<p>확률변수 <img src="https://latex.codecogs.com/png.latex?Y">의 밀도 함수 <img src="https://latex.codecogs.com/png.latex?f(y;%20%5Ctheta)">가 주어지며, 이때 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">는 스칼라 매개변수라고 가정하겠습니다. 또, <img src="https://latex.codecogs.com/png.latex?%5Cell">이 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">에 대해 최소 두 번 미분 가능하다고 가정하면, 단일 관측치 <img src="https://latex.codecogs.com/png.latex?Y">에 대한 로그 가능도(log likelihood) 함수 <img src="https://latex.codecogs.com/png.latex?%5Cell(%5Ctheta;%20Y)">에 대해서 함수의 첫 번째 및 두 번째 도함수는 다음과 같습니다.</p>
<ul>
<li>
<strong>첫 번째 도함수</strong>: <img src="https://latex.codecogs.com/png.latex?%20%5Cell'%20=%20%5Cfrac%7Bd%5Cell%7D%7Bd%5Ctheta%7D%20">
</li>
<li>
<strong>두 번째 도함수</strong>: <img src="https://latex.codecogs.com/png.latex?%20%5Cell''%20=%20%5Cfrac%7Bd%5E2%5Cell%7D%7Bd%5Ctheta%5E2%7D%20">
</li>
</ul>
<p>이때, 이 두 함수들은 다음 관계식이 성립합니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20%5C%7D%20=%200%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft%5B%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20%5C%7D%5E2%20%5Cright%5D%20=%20-E%20%5C%7B%20%5Cell''(%5Ctheta;%20Y)%20%5C%7D%0A"></p>
<p>이를 증명해보겠습니다. 의지를 잃지 않기 위해 위 식들의 의미를 스포하자면, MLE를 통해 모델을 추정할 때 보통 l<strong>og likelihood를 모델의 parameter로 미분한 식이 0(또는 영벡터)이 되도록 하는 parameter를 찾음으로써 이를 수행</strong>하는데, 첫 번째 식은 이 <strong>미분한 식(score function)의 기댓값(평균)이 0이라는 의미</strong>이고, 두 번째 식은 첫 번째 식에서 mean이 0이었으므로 왼쪽항의 제곱 안에 -0을 넣어주면 <img src="https://latex.codecogs.com/png.latex?%0AE%5Cleft%5B%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20-%20E%20%5B%20%5Cell'(%5Ctheta;%20Y)%20%5D%20%5C%7D%5E2%20%5Cright%5D%0A"></p>
<p>가 되어 분산 term이 되고, 따라서 <strong>분산은 이차 도함수의 기댓값의 음수와 같다는 의미</strong>입니다. 이러한 성질들을 이용해서 앞으로 Likelihood 기반의 다양한 모델 추정을 수행할 수 있게 됩니다.<br></p>
<p><strong>(1) Prove</strong> <img src="https://latex.codecogs.com/png.latex?E%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20%5C%7D%20=%200"><strong>.</strong></p>
<p>우선, 확률 분포는 모든 범위에서의 적분 또는 누적합이 1이므로,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A1%20=%20%5Cint%20f(y;%20%5Ctheta)%20dy%0A"></p>
<p>입니다. 이제 양변을 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">에 대해 미분한 후 미분과 적분의 순서를 바꾸면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A0%20=%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20%5Cint%20f(y;%20%5Ctheta)%20dy%20%5C%5C%0A=%5Cint%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20f(y;%20%5Ctheta)%20dy%0A"></p>
<p>입니다. 여기서 수학적 증명 과정에서 굉장히 자주 사용되는 skill <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20f(y;%20%5Ctheta)%20=%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20%5C%7B%20%5Clog%20f(y;%20%5Ctheta)%20%5C%7D%20f(y;%20%5Ctheta)">를 사용하면</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A0%20=%20%5Cint%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20f(y;%20%5Ctheta)%20dy%20%20%5C%5C%0A=%20%5Cint%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20%5C%7B%20%5Clog%20f(y;%20%5Ctheta)%20%5C%7D%20f(y;%20%5Ctheta)%20dy%20%5C%5C%0A=%20%5Cint%20%5Cell'(%5Ctheta;%20Y)%20f(y;%20%5Ctheta)%20dy%20%5C%5C%0A=%20E%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20%5C%7D%0A"> 입니다. 의미를 다시 해석해보면, 어떠한 분포를 따르는 <img src="https://latex.codecogs.com/png.latex?Y">와 이의 매개변수 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">에 대해서, 우리는 MLE를 통해 log likelihood 함수 <img src="https://latex.codecogs.com/png.latex?%5Cell(%5Ctheta;%20Y)">를 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">로 미분하였을 때 0이 나오도록 하는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Ctheta%7D">를 찾음으로써 parameter를 estimate합니다. 위 (1)은 이러한 <img src="https://latex.codecogs.com/png.latex?%5Cell'(%5Ctheta;%20Y)">의 기댓값은 <img src="https://latex.codecogs.com/png.latex?Y">의 분포가 이계도함수가 존재한다면 어떤 분포이건 관계 없이 0임을 보인 것입니다.</p>
<p><strong>(2) Prove</strong> <img src="https://latex.codecogs.com/png.latex?%5Cell''%20=%20%5Cfrac%7Bd%5E2%5Cell%7D%7Bd%5Ctheta%5E2%7D"><strong>.</strong></p>
<p>동일한 논리를 따라 위 식을 한 번 더 미분하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A0%20=%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20%5Cleft%5B%20%5Cint%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20%5C%7B%20%5Clog%20f(Y;%20%5Ctheta)%20%5C%7D%20f(y;%20%5Ctheta)%20dy%20%5Cright%5D%0A"></p>
<p>입니다. 두 함수의 곱 형태의 미분이며 둘 다 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">를 포함하므로 이를 전개하고 마찬가지로 기댓값으로 표기하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A0%20=%20%5Cint%20%5Cfrac%7Bd%5E2%7D%7Bd%5Ctheta%5E2%7D%20%5C%7B%20%5Clog%20f(y;%20%5Ctheta)%20%5C%7D%20f(y;%20%5Ctheta)%20dy%20+%20%5Cint%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20%5C%7B%20%5Clog%20f(y;%20%5Ctheta)%20%5C%7D%20%5Cfrac%7Bd%7D%7Bd%5Ctheta%7D%20f(y;%20%5Ctheta)%20dy%20%5C%5C%0A=%20E%20%5C%7B%20%5Cell''(%5Ctheta;%20Y)%20%5C%7D%20+%20E%20%5Cleft%5B%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20%5C%7D%5E2%20%5Cright%5D%0A"></p>
<p>이고, 따라서 아래 식이 증명되었습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft%5B%20%5C%7B%20%5Cell'(%5Ctheta;%20Y)%20%5C%7D%5E2%20%5Cright%5D%20=%20-E%20%5C%7B%20%5Cell''(%5Ctheta;%20Y)%20%5C%7D%0A"></p>
<p>위 증명 과정에서 미분과 적분 연산자의 교환을 정당화하는 과정의 설명이 생략되었지만 exponential family에선 문제가 없고, Y가 discrete한 경우는 적분을 누적합으로 바꿔주면 된다고 얘기해두며 마무리 하겠습니다. 또한, 위 증명에서는 <img src="https://latex.codecogs.com/png.latex?%5Ctheta">가 <strong>1차원 스칼라 변수</strong>라고 가정했지만, <strong>다차원 매개변수</strong>에 대해서도 동일한 결과가 성립됩니다. 식의 의미를 마지막으로 되짚어보면, log likelihood의 일차 도함수는 기대값이 0이고, 이 일차 도함수의 공분산 행렬은 <strong>이차 도함수 행렬의 기대값의 음수</strong>에 해당합니다. 이 값은 <strong>피셔 정보 행렬(Fisher Information Matrix)</strong>이라고도 불립니다. (함수의 기댓값이라는 말이 어색하게 들릴 수도 있는데, 애초에 모든 랜덤(확률)변수는 어떠한 관측치에 대해서 실수를 output하는 함수임을 되새기면 좋을 것 같습니다.)</p>
</section><section id="exponential-family-성질" class="level3"><h3 class="anchored" data-anchor-id="exponential-family-성질">2.2 Exponential Family 성질</h3>
<hr>
<p>이번에는 위에서 증명한 수식을 통해서 Exponential Family를 소개할 때 언급한 <img src="https://latex.codecogs.com/png.latex?E(Y)%20=%20b'(%5Ctheta)%20=%20%5Cmu">, <img src="https://latex.codecogs.com/png.latex?%5Coperatorname%7Bvar%7D(Y)%20=%20%5Cphi%20b''(%5Ctheta)%20=%20%5Cphi%20V(%5Cmu)">을 증명할 것입니다. Exponential Family distribution은 다음과 같은 일반적인 형식으로 정의됩니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Af(y;%20%5Ctheta,%20%5Cphi)%20=%20%5Cexp%20%5Cleft%5C%7B%20%5Cfrac%7By%5Ctheta%20-%20b(%5Ctheta)%7D%7Ba(%5Cphi)%7D%20+%20c(y,%20%5Cphi)%20%5Cright%5C%7D%0A"> 때문에 log likelihood는 단순하게 아래와 같이 도출됩니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cell(y;%20%5Ctheta,%20%5Cphi)%20=%20%5Cfrac%7By%5Ctheta%20-%20b(%5Ctheta)%7D%7Ba(%5Cphi)%7D%20+%20c(y,%20%5Cphi)%0A"> 이제 이 log likelihood의 derivatives를 계산하면 다음과 같습니다:</p>
<ul>
<li><p><strong>첫 번째 도함수 (Score Function):</strong> <img src="https://latex.codecogs.com/png.latex?%0A%5Cell'%20(y;%20%5Ctheta,%20%5Cphi)%20=%20%5Cfrac%7By%20-%20b'(%5Ctheta)%7D%7Ba(%5Cphi)%7D%0A"></p></li>
<li><p><strong>두 번째 도함수 (Observed Information):</strong> <img src="https://latex.codecogs.com/png.latex?%0A%5Cell''%20(y;%20%5Ctheta,%20%5Cphi)%20=%20%5Cfrac%7B-b''(%5Ctheta)%7D%7Ba(%5Cphi)%7D%0A"></p></li>
</ul>
<p>이 두 함수를 통해서 위에서 유도한 두 공식을 활용하면 다음 두 수식을 얻을 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft%5C%7B%20%5Cfrac%7BY%20-%20b'(%5Ctheta)%7D%7Ba(%5Cphi)%7D%20%5Cright%5C%7D%20=%200%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft%5B%20%5Cleft(%20%5Cfrac%7BY%20-%20b'(%5Ctheta)%7D%7Ba(%5Cphi)%7D%20%5Cright)%5E2%20%5Cright%5D%20=%20%5Cfrac%7Bb''(%5Ctheta)%7D%7Ba(%5Cphi)%7D%0A"> 이때, 식을 잘 보면 첫 번째 식은 결국 <img src="https://latex.codecogs.com/png.latex?E%5BY%20-%20b'(%5Ctheta)%5D%20=%20E%5BY%5D%20-%20b'(%5Ctheta)%20=%200"> 이 되어 <img src="https://latex.codecogs.com/png.latex?E%5C%7B%20Y%20%5C%7D%20=%20b'(%5Ctheta)">을 얻을 수 있고, 두 번째 식에서 <img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft%5B%20%5Cleft(%20%5Cfrac%7BY%20-%20b'(%5Ctheta)%7D%7Ba(%5Cphi)%7D%20%5Cright)%5E2%20%5Cright%5D%20=%20%5Cfrac%7BE%5B(Y%20-%20b'(%5Ctheta))%5E2%5D%7D%7BE%5Ba(%5Cphi)%5E2%5D%7D%20=%20%5Cfrac%7B%5Coperatorname%7BVar%7D(Y)%7D%7Ba(%5Cphi)%5E2%7D%20=%20%5Cfrac%7Bb''(%5Ctheta)%7D%7Ba(%5Cphi)%7D%0A"> 이므로, <img src="https://latex.codecogs.com/png.latex?%5Ctext%7BVar%7D(Y)%20=%20b''(%5Ctheta)%20a(%5Cphi)">임을 보일 수 있습니다.<br></p>
<p>증명한 수식을 다시 한 번 확인하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE(Y)%20=%20b'(%5Ctheta)%20=%20%5Cmu%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AVar(Y)%20=%20a(%5Cphi)%20V(%5Cmu_i)%20=%20a(%5Cphi)b''(%5Ctheta)%0A"></p>
</section><section id="glms-parameter-추정식-유도" class="level3"><h3 class="anchored" data-anchor-id="glms-parameter-추정식-유도">2.3. GLMs’ parameter 추정식 유도</h3>
<hr>
<p>이제 필요한 식이 준비되었으니, 위에서 계속 다루고 있는 log likelihood을 이용해서 MLE estimation으로 GLMs’ parameter을 추정하는 과정을 살펴볼 것입니다. 이때, 추정 과정은 계속 언급한대로 Exponential Family distribution을 따르는 종속변수에 대해서 log likelihood의 model parameter에 대해 미분한 식이 (parameter가 벡터이므로, 좀더 엄밀하게 정의해야 하지만, 의미는 같으니 이렇게 얘기하겠습니다.) 0이 되게 하는 parameter를 찾음으로써 수행되며, <strong>이 때의 함수 (log likelihood의 1차 도함수)를 앞으로는 score function</strong>이라고 부르겠습니다.</p>
<p>우리는 MLE estimation을 통해 여러 Exponential Family distributions에 대해 통일된 estimation algorithm으로 parameter를 추정할 수 있습니다. (이러한 분포 가정 마저 없다면, 3장 GEE에서 보겠지만 분포에 대한 직접적 가정없이 cumulative generating function 등 몇 함수 만으로 Likelihood를 고려하는 Quasi-likelihood Estimation의 개념으로 이어집니다.)</p>
<p>주어진 data가 <img src="https://latex.codecogs.com/png.latex?(y_1,%20...%20,%20y_n)">일 때, 위에서부터 계속 사용해왔던 log-likelihood function은 다음과 같습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Al%20=%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cleft(%20%5Cfrac%7By_i%20%5Ctheta_i%20-%20b(%5Ctheta_i)%7D%7B%5Cphi_i%7D%20+%20c(y_i,%20%5Cphi_i)%20%5Cright)%0A"></p>
<p><strong>지금까지 우리는</strong> <img src="https://latex.codecogs.com/png.latex?%5Ctheta"><strong>로 log likelihood를 다뤘지만, 추정해야 하는 parameter는</strong> <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D"><strong>입니다</strong>. <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">는 벡터이기 때문에 이 중 하나의 파라미터 <img src="https://latex.codecogs.com/png.latex?%5Cbeta_j">에 대해 먼저 log likelihood를 미분해보면 아래와 같은 식이 나옵니다. (<img src="https://latex.codecogs.com/png.latex?%5Ctext%7BVar%7D(y)%20=%20%5Cphi_i%20V(%5Cmu_i)">, <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20=%20%5Cfrac%7B1%7D%7Bg'(%5Cmu_i)%7D">)임은 위에서 보았습니다. g는 link function이었습니다.)</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_j%7D%20=%20s(%5Cbeta_j)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cleft(%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Ctheta%7D%20%5Cright)%20%5Cleft(%5Cfrac%7B%5Cpartial%20%5Ctheta%7D%7B%5Cpartial%20%5Cmu%7D%20%5Cright)%20%5Cleft(%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%20%5Cleft(%5Cfrac%7B%5Cpartial%20%5Ceta%7D%7B%5Cpartial%20%5Cbeta_j%7D%20%5Cright)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cfrac%7By_i%20-%20%5Cmu_i%7D%7B%5Ctext%7BVar%7D(y_i)%7D%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%20x_%7Bij%7D,%20%5Cquad%20%5Ctext%7Bor%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Csum_%7Bi=1%7D%5E%7Bn%7D%20%5Cfrac%7By_i%20-%20%5Cmu_i%7D%7B%5Cphi_i%20V(%5Cmu_i)%7D%20%5Ctimes%20%5Cfrac%7Bx_%7Bij%7D%7D%7Bg'(%5Cmu_i)%7D%20=%200%0A"></p>
<p>식이 혼란스러울 수 있는데, 이는 단지 chain rule을 이용해서 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">에 대한 <img src="https://latex.codecogs.com/png.latex?%5Cell">의 기울기를 구하는 과정이며, 위에서 GLM을 구성하는 과정을 차근차근 복기하면 각각의 변화율은 다음과 같이 구할 수 있기 때문에 최종 식을 얻을 수 있었음을 알 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Ctheta%7D%20=%0A%5Cfrac%7By%20-%20b'(%5Ctheta)%7D%7Ba(%5Cphi)%7D%20=%0A%5Cfrac%7By%20-%20%5Cmu%7D%7Ba(%5Cphi)%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20%5Ctheta%7D%7B%5Cpartial%20%5Cmu%7D%20=%0A%5Cfrac%7B1%7D%7Bb''(%5Ctheta)%7D%20=%0A%5Cfrac%7B1%7D%7BV(%5Cmu)%7D%20=%0A%5Cfrac%7Ba(%5Cphi)%7D%7B%5Ctext%7BVar%7D(y)%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20%5Ceta%7D%7B%5Cpartial%20%5Cbeta_j%7D%20=%20x_%7Bij%7D%0A"></p>
<p>이제 이 score function의 음의 미분(또는 분산)의 기댓값을 전개하면 다음과 같습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A-%20E%20%5Cleft(%20%5Cfrac%7B%5Cpartial%5E2%20l%7D%7B%5Cpartial%20%5Cbeta_j%20%5Cpartial%20%5Cbeta_k%7D%0A%5Cright)%20=%20E%20%5Cleft%5B%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_j%7D%20%5Cright)%0A%5Cleft(%20%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_k%7D%20%5Cright)%20%5Cright%5D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20E%20%5Cleft%5B%20%5Cleft(%20%5Cfrac%7By%20-%20%5Cmu%7D%7B%5Ctext%7BVar%7D(y)%7D%20%5Cright)%5E2%0A%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%5E2%0Ax_%7Bij%7D%20x_%7Bik%7D%20%5Cright%5D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20E%20%5Cleft%5B%20%5Cfrac%7B%5Ctext%7BVar%7D(y)%7D%7B%5Ctext%7BVar%7D(y)%5E2%7D%0A%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%5E2%0Ax_%7Bij%7D%20x_%7Bik%7D%20%5Cright%5D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%20%5Cleft(%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%0A%5Cright)%5E2%20x_%7Bij%7D%20x_%7Bik%7D%0A"></p>
<p>위 term은 <strong>Fisher Information matrix</strong>라고도 부르며, 이 term이 분산의 기댓값인 이유를 생각해보면, 이전에 구한 derivatives of log likelihood의 성질들에 의해 <img src="https://latex.codecogs.com/png.latex?%5Cell">의 negative 2차 도함수의 기댓값은 1차 도함수의 기댓값의 square와 같고, 이 1차 도함수의 기댓값이 0이므로 이는 분산과 같습니다. 정리하자면, <strong>score function의 미분식이 score function의 분산과 기댓값이 같으므로, 미분을 직접하는 대신 분산으로 근사 후 식을 전개한 것이며, 이 때 근사한 이 Matrix를 Fisher Information matrix라고 부릅니다.</strong></p>
<p>이들을 한 번에 벡터와 행렬 연산으로 표현하면 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta%7D%20=%20X%5E%5Ctop%20A%20(y%20-%20%5Cmu)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft(%20%5Cfrac%7B%5Cpartial%5E2%20l%7D%7B%5Cpartial%20%5Cbeta%20%5Cpartial%20%5Cbeta%5ET%7D%20%5Cright)%20=%0A-X%5ET%20W%20X%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0Awhere,%20W%20%5C;%20%5Ctext%7Bis%20diagonal%20matrix%20comes%20from%20%7D%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%0A%5Cleft(%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%5E2,%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7Band%20%7DA%20%5C;%20%5Ctext%7Bis%20diagonal%20matrix%20comes%20from%20%7D%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%0A%5Cleft(%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright).%0A"></p>
<p>이제 우리는 score function와 그 미분, 또는 log likelihood의 1차 도함수와 (negative) 2차 도함수의 추정식을 얻었습니다. 사실, <img src="https://latex.codecogs.com/png.latex?X%5E%5Ctop%20A%20(y%20-%20%5Cmu)"><strong>가 0이 되도록 하는 parameter</strong> <img src="https://latex.codecogs.com/png.latex?%5Cbeta"><strong>만 찾으면 parameter 추정이 끝나지만, 이는 closed-form solution이 존재하지 않기 때문에 그 미분(2차 도함수)식을 이용해 근사적으로 구할 수 있는 알고리즘을 최종적으로 다룰 것입니다.</strong> (물론 후에 보겠지만 이 Fisher Information matrix는 분산과도 관련이 있습니다.) 즉, 우리는 <em>추정식을 유도하는 것은 완성했지만, 실제로 알고리즘을 설계하여 어떻게 이를 추정할 지에 대해서는 모르는 상태</em>이고, 때문에 최종적으로 GLM을 제안한 학자가 소개하였으며 대부분의 패키지에서 이 GLM을 estimate하기 위해 사용하고 있는 method인 IRLS(Iteratively Reweighted Least Squares) Algorithm을 살펴볼 것입니다. (negative) 2차 도함수의 추정식(or Fisher Matrix) <img src="https://latex.codecogs.com/png.latex?X%5ET%20W%20X">을 유도한 이유는, 이 알고리즘에서 필요로 하기 때문이고, <strong>위 식에서</strong> <img src="https://latex.codecogs.com/png.latex?W,%20A"><strong>는 식이 복잡해보이지만, 그저 observations(data) 하나 당 GLM 모델의 구성요소를 통해 determinant하게 미리 계산되어 대각성분으로 각각 들어가는 term임</strong>을 명심하시면 좋을 것 같습니다. (이전 설명에서, 종속변수의 분포로 Exponential Family 중 특정 분포가 정해지면, <strong>이에 따라 Link function(Canonical link), Variance function가 정해져 모델이 특정된다고 설명드린 적이 있고, 위</strong> <img src="https://latex.codecogs.com/png.latex?W,%20A"><strong>모두 이 두 함수로 이루어진 식이기 때문에 관측치마다 각각 넣으면 determinant하게 하나의 값이 나오는 식인 것입니다.</strong>)</p>
</section><section id="glms-parameter-추정-irls" class="level3"><h3 class="anchored" data-anchor-id="glms-parameter-추정-irls">2.4. GLMs’ parameter 추정 (IRLS)</h3>
<hr>
<p>위에서 언급하였듯, GLM의 MLE estimation은 <strong>비선형 최적화 문제</strong>이기 때문에 공통된 framework에서 사용할 수 있는 closed-form solution이 존재하지 않으며, 대신 여러 최적화 방법을 사용할 수 있습니다. <strong>뉴턴-랩슨 방법(Newton-Raphson Method)</strong>은 2차 도함수(Hessian Matrix)를 사용하여 score function을 수렴시키지만, Hessian Matrix <img src="https://latex.codecogs.com/png.latex?H%20=%5Cfrac%7B%5Cpartial%5E2l%7D%7B%5Cpartial%20%5Cbeta%20%5Cpartial%20%5Cbeta%5ET%7D">를 직접 구해야 하고, 이는 계산이 복잡하여 computation cost가 큽니다. <strong>Fisher Scoring</strong>은 Newton Method에서 Hessian Matrix 대신 이를 근사하는 <strong>Fisher Information Matrix를</strong> 사용합니다. 이는 이전에 Derivatives of log likelihood 의 성질이랑 추정식 유도에서 모두 보았던대로 2차 도함수가 1차 도함수의 분산(또는 제곱)와 기댓값이 같다는 수학적 성질을 토대로 <img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft(%0A%5Cfrac%7B%5Cpartial%5E2%20l%7D%7B%5Cpartial%20%5Cbeta%20%5Cpartial%20%5Cbeta%5ET%7D%20%5Cright)%20=%20E%0A%5Cleft%5B%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_j%7D%20%5Cright)%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_k%7D%20%5Cright)%20%5Cright%5D%0A=%20X%5ET%20W%20X%5C%0A"> 가 만족함을 확인하였기 때문에, MLE추정에서 <strong>Newton-Raphson Method</strong>보다 computation cost가 합리적인 method라고 생각해 볼 수 있습니다. 이외에도 <strong>경사 하강법(Gradient Descent)</strong> 알고리즘은 <strong>1차 도함수(Gradient)만 사용</strong>하여 특정 값만큼 조금씩 점진적으로 parameter를 움직여 최적점을 찾는 방법입니다. 모델이 매우 복잡해서 2차 도함수를 계산하기 힘든 딥러닝에서는 많은 경우 이를 발전시킨 여러 methods로 iterativaly하게 parameter를 추정합니다. (이렇게 iterative하게 model’s parameter를 움직이면서 추정하는 과정이 AI에서 얘기하는 learning입니다.)</p>
<p>위에서 얘기한 <strong>IRLS(Iteratively Reweighted Least Squares) Algorithm</strong>는 이 <strong>Fisher Scoring</strong>의 알고리즘적 변형으로, Fisher Scoring의 식을 <strong>가중 최소제곱(Weighted Least Squares)</strong> 문제로 치환하여, 이 문제에서 사용하는 IRLS 알고리즘으로 GLM의 parameter 해를 구하는 method입니다. 우선 Newton-Raphson Method, Fisher Scoring에 대한 이야기를 간단하게 하고, 자세하게 어떻게 IRLS가 GLM의 parameter를 추정하는지 보겠습니다.</p>
<section id="newton-raphson-method-fisher-scoring" class="level4"><h4 class="anchored" data-anchor-id="newton-raphson-method-fisher-scoring">Newton-Raphson Method &amp; Fisher Scoring</h4>
<hr>
<p>GLM (Generalized Linear Model)의 파라미터 추정을 위한 최적화 과정은 우선 log likelihood function의 최대화를 목적으로 합니다. 이때, <strong>Newton-Raphson Method</strong>와 그 변형인 <strong>Fisher Scoring</strong>은 모두 이를 위한 알고리즘입니다.</p>
<p>Newton-Raphson 방법은 다음과 같은 일반적인 업데이트 식을 갖습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cboldsymbol%7B%5Cbeta%7D%5E%7B(t+1)%7D%20=%20%5Cboldsymbol%7B%5Cbeta%7D%5E%7B(t)%7D%20-%20%5Cleft%5B%5Cfrac%7B%5Cpartial%5E2%20%5Cell%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%20%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%5E%5Ctop%7D%5Cright%5D%5E%7B-1%7D%20%5Cfrac%7B%5Cpartial%20%5Cell%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%7D,%0A"> 이후의 method들은 모두 이 method에서 기인하므로, 위 식의 의미를 이해하는 것은 매우 중요합니다. 위 식이 어떻게 안정적으로 <img src="https://latex.codecogs.com/png.latex?%5Cboldsymbol%7B%5Cbeta%7D">를 수렴시킬 수 있는지 2차 테일러 전개를 통해 수식적으로 좀 더 명확히 볼 수 있지만, 여기서는 좀더 직관적으로 가볍게 이해해보겠습니다.</p>
<p>Newton-Raphson 방법은 어떠한 함수 <img src="https://latex.codecogs.com/png.latex?f(x)">에 대해서 함수의 해, 즉 <img src="https://latex.codecogs.com/png.latex?f(x)%20=%200">을 만족하는 <img src="https://latex.codecogs.com/png.latex?x">를 찾기 위한 반복적인 근사 방법입니다.(informal한 증명이므로 1-dimension case로 보겠습니다.) 이 방법은 현재 점에서 함수의 접선을 그려 <img src="https://latex.codecogs.com/png.latex?x">축과 만나는 점을 다음 근사해로 사용합니다. 예를 들어, 초기값 <img src="https://latex.codecogs.com/png.latex?x_0">에서 접선을 그리면 그 접선의 방정식은 <img src="https://latex.codecogs.com/png.latex?y%20=%20f'(x_0)(x%20-%20x_0)%20+%20f(x_0)">입니다. 이 접선이 <img src="https://latex.codecogs.com/png.latex?x"> 축과 만나는 점은 <img src="https://latex.codecogs.com/png.latex?y%20=%200">일 때이므로, 이를 대입하여 접선이 0이 되는 점을 구해보면 <img src="https://latex.codecogs.com/png.latex?x_1%20=%20x_0%20-%20%5Cfrac%7Bf(x_0)%7D%7Bf'(x_0)%7D">가 됩니다. 그러나 접선은 함수의 선형 approximation이므로 이 접선이 0이 되는 점 <img src="https://latex.codecogs.com/png.latex?x_1">이 실제 함수에서도 곧바로 0이 되지는 않습니다. 따라서 이 과정을 반복하여 특정 단계에서 <img src="https://latex.codecogs.com/png.latex?f(x_n)">이 0에 충분히 가까워지면, <img src="https://latex.codecogs.com/png.latex?x_n">을 근사해로 채택합니다.</p>
<p>이 방법이 작동하는 이유는 접선의 기울기 <img src="https://latex.codecogs.com/png.latex?f'(x_n)">이 함수의 곡률을 반영하기 때문입니다. 곡률이 클수록(기울기가 가파를수록) 업데이트의 크기가 작아지고, 곡률이 작을수록 업데이트의 크기가 커집니다. 또한, Newton-Raphson 방법은 <strong>2차 수렴(Quadratic Convergence)</strong> 속도를 가집니다. 이는 오차가 반복마다 제곱으로 줄어들기 때문에 매우 빠르게 해에 수렴한다는 의미입니다.</p>
<p>이정도로 간단하게 Newton-Raphson method를 이해할 수 있고, 다시 돌아와서 위 식에서는 multi-dimention 상황에서 해를 찾고 싶은 함수가 score function, 즉 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20%20%5Cell%7D%7B%5Cpartial%20%5Cbeta%7D">인 경우이기 때문에 위처럼 식이 구성되었다는 것을 알 수 있습니다.(f의 미분이 분모로 들어간 term은 행렬에서 역행렬 -1과 같은 의미라고 보시면 됩니다.) 첨언하자면, 이 경우 Newton-Raphson method는 두 가지 중요 조건이 붙는데, 언급만 하자면 <strong>Hessian matrix가 Positive-definite (볼록) 해야 하며, 초기값이 최적점에 충분히 가까워야 합니다.</strong></p>
<p><strong>Fisher Scoring</strong>은 Hessian 행렬 대신 Fisher Information 행렬 <img src="https://latex.codecogs.com/png.latex?%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D)">를 사용하여 다음과 같이 업데이트합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cboldsymbol%7B%5Cbeta%7D%5E%7B(t+1)%7D%20=%20%5Cboldsymbol%7B%5Cbeta%7D%5E%7B(t)%7D%20+%20%5Cleft(%20%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D%5E%7B(t)%7D)%20%5Cright)%5E%7B-1%7D%20%5Cmathbf%7BS%7D(%5Cboldsymbol%7B%5Cbeta%7D%5E%7B(t)%7D),%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BS%7D(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20%5Cfrac%7B%5Cpartial%20%5Cell%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%7D%0A"> 는 Score function으로 구한 (Fisher) score 벡터이고, <img src="https://latex.codecogs.com/png.latex?%0A%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D)%20=%20E%0A%5Cleft%5B%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_j%7D%20%5Cright)%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta_k%7D%20%5Cright)%20%5Cright%5D%20=%20E%5Cleft%5B-%5Cfrac%7B%5Cpartial%5E2%20%5Cell%7D%7B%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%20%5Cpartial%20%5Cboldsymbol%7B%5Cbeta%7D%5E%5Ctop%7D%5Cright%5D%0A"> 는 <strong>Fisher information matrix</strong>입니다.</p>
<p>즉, Fisher Scoring 방법은 <strong>뉴턴-랩슨 방법(Newton-Raphson Method)</strong>에서 Hessian Matrix <img src="https://latex.codecogs.com/png.latex?H%20=%5Cfrac%7B%5Cpartial%5E2l%7D%7B%5Cpartial%20%5Cbeta%20%5Cpartial%20%5Cbeta%5ET%7D"> 를 사용하는 대신, <strong>Fisher information matrix를 사용해서 업데이트를 수행함</strong>으로써 parameter를 estimation하는 매커니즘입니다. <strong>IRLS</strong>는 GLM에서 이 Fisher Scoring와 거의 일치하다 봐도 무방하며, <strong>단순히 위에서 추정한</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BS%7D(%5Cboldsymbol%7B%5Cbeta%7D)"><strong>와</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathcal%7BI%7D(%5Cboldsymbol%7B%5Cbeta%7D)"><strong>를 Fisher Scoring 공식에 넣으면 weighted least squares problme(가중 최소제곱 문제)와 완전히 유사해지기 때문에</strong>, 이 문제를 해결하는 방식으로 parameter를 추정한다는 의미라고 생각하시면 될 것 같습니다. 좀 더 수식과 같이 자세하게 설명드리겠습니다.</p>
</section><section id="irls-iteratively-reweighted-least-squares-algorithm" class="level4"><h4 class="anchored" data-anchor-id="irls-iteratively-reweighted-least-squares-algorithm">IRLS (Iteratively Reweighted Least Squares) Algorithm</h4>
<hr>
<p>앞서, 2.3.에서 log likelihood의 <strong>gradient(1차 도함수, Score function)</strong>와 <strong>expected Hessian(Fisher Information matrix)</strong>가 각각</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cpartial%20l%7D%7B%5Cpartial%20%5Cbeta%7D%20=%20X%5ET%20A%20(y%20-%20%5Cmu)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%20%5Cleft(%20%5Cfrac%7B%5Cpartial%5E2%20l%7D%7B%5Cpartial%20%5Cbeta%20%5Cpartial%20%5Cbeta%5ET%7D%20%5Cright)%20=%20X%5ET%20W%20X%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0Awhere,%20%5Cquad%20A%20=%20%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%20%5C;%20and%20%5Cquad%20W%20=%20%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%5E2%0A"></p>
<p>임을 보았습니다. 이 결과들을 Fisher Scoring method의 식에 대입하면,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbeta%5E%7B(t+1)%7D%20=%20%5Cbeta%5E%7B(t)%7D%20+%20%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5E%7B-1%7D%20X%5ET%20A%20(y%20-%20%5Cmu)%0A"></p>
<p>이고, 이 equation은 <strong>가중 최소제곱 문제(Weighted Least Squares, WLS)</strong>의 parameter 추정식과 일치하기 때문에 비선형 GLM의 parameter 추정을 WLS problem으로 치환할 수 있음을 알 수 있습니다.</p>
<p>위의 업데이트 식은 아래 <strong>working response</strong> <img src="https://latex.codecogs.com/png.latex?z">를 정의하면</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Az%20=%20X%20%5Cbeta%5E%7B(t)%7D%20+%20W%5E%7B-1%7D%20A%20(y%20-%20%5Cmu),%0A"></p>
<p>아래와 같이 표현할 수 있습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbeta%5E%7B(t+1)%7D%20=%20%5Cbeta%5E%7B(t)%7D%20+%20%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5E%7B-1%7D%20X%5ET%20A%20(y%20-%20%5Cmu)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5E%7B-1%7D%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5Cbeta%5E%7B(t)%7D%20+%20%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5E%7B-1%7D%20X%5ET%20W%20W%5E%7B-1%7D%20A%20(y%20-%20%5Cmu)%0A"> <img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5E%7B-1%7DX%5ET%20W%20%5Cleft(%20X%20%5Cbeta%5E%7B(t)%7D%20+%20W%5E%7B-1%7D%20A%20(y%20-%20%5Cmu)%20%5Cright)%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A=%20%5Cleft(%20X%5ET%20W%20X%20%5Cright)%5E%7B-1%7D%20X%5ET%20W%20z%0A"></p>
<p>또 강조하지만, 이는 가중치 행렬 <img src="https://latex.codecogs.com/png.latex?W">에 따라 각 관측치의 기여도를 달리하는 선형 회귀 문제(가중 최소제곱 문제)의 정규방정식과 동일합니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A(X%5ET%20W%20X)%20%5Cbeta%20=%20X%5ET%20W%20z.%0A"></p>
<p>따라서 해당 정규방정식의 <img src="https://latex.codecogs.com/png.latex?%5Cbeta">를 가중 최소제곱 문제 방식으로 풀어냄으로써 추정치 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(t+1)%7D">를 구할 수 있으며, 이는 현재 단계의 추정치 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(t)%7D">에서의 예측값과 오차를 반영한 새로운 업데이트가 됩니다.</p>
</section><section id="irls-구체적-절차" class="level4"><h4 class="anchored" data-anchor-id="irls-구체적-절차">IRLS 구체적 절차</h4>
<hr>
<p>정리하자면, <strong>IRLS(Iteratively Reweighted Least Squares)</strong> 알고리즘은 위의 아이디어를 바탕으로 GLM의 최대우도추정 문제를 반복적으로 가중 최소제곱 문제로 전환하여 해결합니다. 구체적인 단계는 다음과 같습니다:</p>
<p><strong>(1) 초기화</strong>: 초기 파라미터 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(0)%7D">를 설정합니다.</p>
<p><strong>(2) 현재 단계 계산</strong>:</p>
<p><strong>(2.1)</strong> <strong>예측값 계산</strong>: 현재 추정치 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(t)%7D">을 이용하여 선형 예측치 <img src="https://latex.codecogs.com/png.latex?%5Ceta%5E%7B(t)%7D%20=%20X%20%5Cbeta%5E%7B(t)%7D">를 구하고, link 함수의 역함수를 통해 <img src="https://latex.codecogs.com/png.latex?%5Cmu%5E%7B(t)%7D%20=%20g%5E%7B-1%7D(%5Ceta%5E%7B(t)%7D)">를 계산합니다.</p>
<p><strong>(2.2)</strong> <strong>가중치 및 보조 행렬 계산</strong>: 정의한 식에 따라</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AA%5E%7B(t)%7D%20=%20%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%5E%7B(t)%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0AW%5E%7B(t)%7D%20=%20%5Cfrac%7B1%7D%7B%5Ctext%7BVar%7D(y)%7D%20%5Cleft(%20%5Cfrac%7B%5Cpartial%20%5Cmu%7D%7B%5Cpartial%20%5Ceta%7D%20%5Cright)%5E%7B(t)%202%7D%0A"></p>
<p>을 계산합니다.</p>
<p><strong>(2.3)</strong> <strong>Working Response 구성</strong>:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Az%5E%7B(t)%7D%20=%20X%20%5Cbeta%5E%7B(t)%7D%20+%20%5Cleft(%20W%5E%7B(t)%7D%20%5Cright)%5E%7B-1%7D%20A%5E%7B(t)%7D%20(y%20-%20%5Cmu%5E%7B(t)%7D).%0A"></p>
<p><strong>(3) 가중 최소제곱 문제 해결</strong>:</p>
<p>위의 working response와 가중치 행렬을 사용하여 정규방정식</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A(X%5ET%20W%5E%7B(t)%7D%20X)%20%5Cbeta%5E%7B(t+1)%7D%20=%20X%5ET%20W%5E%7B(t)%7D%20z%5E%7B(t)%7D%0A"> 을 풀어 새로운 추정치 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(t+1)%7D">를 구합니다.</p>
<p><strong>(4) 수렴 판단 및 반복</strong>:</p>
<p><img src="https://latex.codecogs.com/png.latex?%7C%7C%5Cbeta%5E%7B(t+1)%7D%20-%20%5Cbeta%5E%7B(t)%7D%7C%7C">(L1 norm, 쉽게는 절댓값)가 미리 설정한 임계값 이하가 될 때까지 2번과 3번의 단계를 반복하고, 수렴이 되었다면 이 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(t+1)%7D"> 값이 GLM의 parameter에 대한 IRLS의 최종 Estimation 결과입니다. 결국, Fisher Scoring에서 대입한 수식이 결국 가중 최소제곱 문제로 귀착됨을 통해, <strong>IRLS 알고리즘</strong>은 각 반복마다 선형 회귀 문제와 유사한 방식으로 parameter를 업데이트합니다.</p>
<p>이 IRLS의 소프트웨어 구현에 대한 첨언을 하자면, 보통 <strong>IRLS</strong>에서는 각 단계마다 위 3단계와 같이 아래 정규방정식을 풉니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A(X%5ET%20W%20X)%20%5Cbeta%20=%20X%5ET%20W%20z.%0A"></p>
<p>이때 직접 <img src="https://latex.codecogs.com/png.latex?(X%5ET%20W%20X)%5E%7B-1%7D">를 구해 업데이트하는 방법은 계산적으로 불안정할 수 있습니다. 특히, 데이터의 규모가 크거나 <img src="https://latex.codecogs.com/png.latex?X"> 행렬이 <strong>ill-conditioned(조건수가 열악한)</strong>인 경우에는 직접 역행렬을 계산하는 과정에서 수치적인 문제가 발생할 위험이 큽니다. 따라서, 구현의 영역이기 때문에 더이상 나열하지는 않겠지만 실제는 역행렬을 direct하게 구하는 대신, <strong>QR 분해</strong>나 <strong>Cholesky 분해</strong> 같은 선형대수 기법을 활용하여 안정적으로 선형 시스템을 풀 수 있습니다.</p>
</section></section><section id="glms-parameter-variance" class="level3"><h3 class="anchored" data-anchor-id="glms-parameter-variance">2.5. GLMs’ parameter Variance</h3>
<hr>
<p>앞서 <strong>IRLS(Iteratively Reweighted Least Squares)</strong> 알고리즘으로 GLM의 파라미터를 추정하는 과정을 살펴보았습니다. 이때 우리는 MLE(최대우도추정)을 IRLS(반복적으로 <strong>가중 최소제곱</strong>)문제로 전환하는 과정을 거쳤는데, 최종적으로 구해지는 추정치 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">의 분산에 대한 추정까지 마쳐야 유의성 검정 등의 분석을 수행할 수 있을 것입니다. GLM에서 최대우도추정(MLE)을 사용해 얻은 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">는,&nbsp;이론적으로 위에서 언급한 <strong>Fisher 정보 행렬</strong>(Fisher information matrix)의 역행렬로써 구할 수 있습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D%5Cbigl(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5Cbigr)%20=%20%5Cbigl(%5Cmathbf%7BI%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%5Cbigr)%5E%7B-1%7D,%20"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BI%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)">는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D">에서의 (observed 혹은 expected) Fisher 정보 행렬입니다.(3장에서 이 이유에 대해 살펴볼 것입니다.) 모델의 parameter를 추정하는 과정에서, 각 관측치 <img src="https://latex.codecogs.com/png.latex?i">에 대해 <img src="https://latex.codecogs.com/png.latex?%5Cmathrm%7BVar%7D(y_i)">와 <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20%5Cmu_i%7D%7B%5Cpartial%20%5Ceta_i%7D">가 포함된 특정 가중치 <img src="https://latex.codecogs.com/png.latex?W%5E%7B(t)%7D">가 등장하였었고, 반복(step)마다 업데이트되는 정규방정식을 풀어감으로써 추정치 <img src="https://latex.codecogs.com/png.latex?%5Cbeta%5E%7B(t+1)%7D">를 얻었습니다. 이후 최종 수렴 시점(<img src="https://latex.codecogs.com/png.latex?t%20%5Cto%20%5Cinfty">)에서, 우리는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%20=%20%5Cbeta%5E%7B(%5Cinfty)%7D">에 도달하게 되고, 이 시점에서 계산된 <strong>Hessian(또는 Fisher information) matrix</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D">에 대해서, 이 행렬의 역수가 분산이 되는 것입니다.</p>
<p>즉, 실제 계산 시에는 아래와 같은 모양이 됩니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D%5Cbigl(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D%5Cbigr)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D)%5E%7B-1%7D,%20"></p>
<p>왜 최종적으로 추정된 모델의 분산이 수렴된 time step에서의 Fisher information matrix로 추정할 수 있는지에 대해서는 3장의 M-estimation에서 증명할 것입니다. 여기서 미리 강조할 것은, 위에서 얻는 분산의 추정값은 GLM에서의 <strong>기본 가정들이었던</strong> 독립성, 분산 함수의 형태 등이 성립한다는 조건 위에서 도출된 것이므로, 이러한 기본 가정이 어느 정도 엄격히 맞아떨어지는 상황(정확한 포아송 분포를 따르는 count data, 명시적으로 독립적인 개별 관측치 등)이라면 괜찮지만, 현실의 data에서는 <strong>이질분산</strong>, <strong>클러스터 내 상관</strong>, <strong>과산포(overdispersion)</strong> 등으로 인해 이 기본 가정들이 깨질 수 있습니다. 다행히도, GLM에서도 모델이 consist할 때(GLM의 위로부터 추정된 parameter 자체는 consist합니다.) <strong>HC(Heteroskedasticity-Consistent) se</strong>와, 아래에서 clustered data에서 고려할 수 있는 버전인 <strong>Cluster-robust standard errors</strong>를 사용하여 더욱 robust하게 분산을 추정할 수 있습니다. 때문에 아래에서는 Cluster-robust se를 OLS 버전으로 소개드린 후, GLM에서 사용하기 위해 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BW%7D%7D"> 행렬 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BA%7D%7D"> 행렬 등의 구조가 어떻게 수식적으로 첨가되어 LM(Linear Model)에서의 식과는 살짝 다른 모습을 취하게 되는지 보고 마치겠습니다.<br></p>
</section></section><section id="cluster-robust-standard-errors" class="level2"><h2 class="anchored" data-anchor-id="cluster-robust-standard-errors">3. Cluster-Robust Standard Errors</h2>
<section id="clustered-data-정의" class="level3"><h3 class="anchored" data-anchor-id="clustered-data-정의">3.1. Clustered Data 정의</h3>
<hr>
<p><strong>Clustered data</strong>란 데이터 내에서 동일 그룹에 속하는 관측치들이 상관관계를 가지는 경우를 의미합니다. 예를 들어, 한 환자의 여러 진료 기록이 서로 상관되어 있을 수 있습니다. 이 때, cluster간에는 상관관계가 없고 cluster 내의 데이터들은 상관관계가 있다는 가정하에 Cluster-robust standard errors나 3장의 GEE, GLMM model이 개발되었습니다. 의료 분석 상황의 예시로는 대표적으로 데이터의 각 환자 당 여러 시간 또는 주기에 걸쳐 측정한 데이터, 여러 학교나 병원과 같은 단체에서 얻은 데이터들을 한 번에 고려하는 경우가 있을 것입니다. 또한, cluster간에 상관관계가 있거나 cluster 안에 cluster가 있는 hierarchical의 경우도 있지만, 이에 대한 공식들은 위에서 고려하는 1차적인 상황을 이해하면 쉽게 이해할 수 있으며, 의료 분석에서 고려하는 피험자 내 관측치 간 상관관계, 병원 내 관측치 간 상관관계 등을 고려해야 하는 상황은 이번 블로그에서 이야기 할 1차적인 clustered 상황임을 알아두시면 좋을 것 같습니다. 이 observations간의 상관관계에 대한 이야기와, 이때 사용해야 하는 Regression Models에 대한 설명은 3장에서 GEE, GLMM과 함께 더욱 자세하게 다뤄볼 예정입니다.</p>
<p>확실한 것은, 비선형 분포를 추정할 수 있는 GLM이나, OLS에서 안정적인 parameter 분산 추정 method였던 HC(Heteroskedasticity-Consistent) 표준오차는 관측치 간의 독립성을 가정하였었고, 이는 <strong>위와 같은 data를 다룰 때에는 깨져야 하는 가정</strong>이라는 것입니다. 이제 설명드릴 <strong>Cluster-robust standard errors</strong>는 HC se와 형태가 매우 비슷하며, 같은 철학으로 <strong>clustered data에서 robust한 모델 분산 추정 method</strong>입니다. 이때 기억하셔야 할 부분은, 1장에서는 HC se의 안정성을 Linear Regression에 대해서 고려하였고 R의 sandwich 패키지를 통해 구현할 수 있음을 보았는데, 이번 장에서 다루고 있는 <strong>GLM에서도 이 HC se, Cluster-robust se를 모두 사용할 수 있다</strong>는 것입니다. 이때 식이 LM과 GLM에서 살짝 다른데, 우선 Linear Model에서의 Cluster-robust se에 대해서 설명드리고, GLM에서는 무엇이 다른지 보겠습니다.</p>
<p>실제 소프트웨어의 구현에 대해서 첨언하자면, R의 sandwich 패키지나 대부분의 패캐지에서는 이 두 robust 분산 추정의 계산 및 검정을 LM, GLM 모두에 사용 가능하고, 이 패키지들은 들어오는 모델의 객체가 LM, GLM임을 분류한 뒤 각각에 맞는 살짝 변형된 식으로 추정한다고 생각하시면 될 것 같습니다.</p>
</section><section id="cluster-robust-standard-errors-정의-및-수학적-표현" class="level3"><h3 class="anchored" data-anchor-id="cluster-robust-standard-errors-정의-및-수학적-표현">3.2. Cluster-robust standard errors 정의 및 수학적 표현</h3>
<hr>
<p>Cluster-robust standard errors는 <strong>클러스터 내 상관관계</strong>를 고려하여 분산을 추정합니다. 이를 통해 클러스터 간 독립성은 유지하되, 클러스터 내 관측치 간 상관관계가 존재할 때도 일관된 추정치를 제공합니다. LM에서 Cluster-robust standard error를 구하는 식은 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20%5Cmathbf%7BX%7D_g%5E%5Ctop%20%5Cmathbf%7B%5Chat%7Bu%7D%7D_g%20%5Cmathbf%7B%5Chat%7Bu%7D%7D_g%5E%5Ctop%20%5Cmathbf%7BX%7D_g%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>위 식에서 <img src="https://latex.codecogs.com/png.latex?g">는 클러스터 인덱스, <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7B%5Chat%7Bu%7D%7D_g">는 클러스터 <img src="https://latex.codecogs.com/png.latex?g">의 잔차 벡터입니다.이 식은 1장에서의 HC0과 아주 유사하며, 가운데 meat항 (두 <img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D"> 사이에 있는 항)만 달라졌음을 알 수 있습니다. 이는 실제로 HC0에서, 위에서 설명드린 가정인 1차적 clustered 구조(클러스터 간 독립성을 가정하지만, 클러스터 내 관측치들 간 상관관계는 허용)를 고려해서 <img src="https://latex.codecogs.com/png.latex?%5CPhi"> 항만 바뀌었음을 짐작해볼 수 있습니다. 이제 이 Cluster-robust standard errors의 철학에 대해서 구체적으로 살펴보겠습니다.</p>
</section><section id="cluster-robust-standard-errors-수학적-표현" class="level3"><h3 class="anchored" data-anchor-id="cluster-robust-standard-errors-수학적-표현">3.3. Cluster-robust standard errors 수학적 표현</h3>
<hr>
<p>LM에서는 이전에 봤던대로 parameter의 분산을 유도하면 다음과 같습니다:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5CPhi%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5CPhi">는 오차항의 공분산 행렬을 나타냈었습니다. HC0 (Heteroskedasticity-Consistent 0)에서는 모든 관측치가 서로 독립임을 가정하였기 때문에 (Heteroskedasticity를 고려하였지 dependent case를 고려하지는 않았었습니다.) 이에 따라 <img src="https://latex.codecogs.com/png.latex?%5CPhi">는 대각행렬로 표현되며,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CPhi_%7B%5Ctext%7BHC0%7D%7D%20=%20%5Coperatorname%7Bdiag%7D(%5Csigma_1%5E2,%20%5Csigma_2%5E2,%20%5Cdots,%20%5Csigma_n%5E2)%0A"></p>
<p>결과적으로 분산 추정량은 개별 관측치에 대해아래와 같이 계산하였었습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D_%7B%5Ctext%7BHC0%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7Bx%7D_i%20%5Chat%7Bu%7D_i%5E2%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"> 위 식은 1장의 HC0 식과 같은 식입니다. 표현이 어색하다고 느끼시는 분을 위해 이전에 사용한 식을 가져오면 <img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D_%7B%5Ctext%7BHC0%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5Coperatorname%7Bdiag%7D(e_i%5E2)%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>이며, 위에서부터 얘기하고 있는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bu%7D">는 이 error term <img src="https://latex.codecogs.com/png.latex?e">와 비슷한(잔차이기 때문에 사실 의미는 다릅니다) 의미입니다. HC0에서는 각 관측치만을 고려하기 때문에 <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bu%7D_i">는 <img src="https://latex.codecogs.com/png.latex?e_i">와 같고, 길이가 1인 벡터, 즉 scalar이기 때문에 제곱을 사용하였지만 위 cluster-robust 식에서 사용한 <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bu%7D_g">는 클러스터 g에 해당하는 모든 관측치를 한 줄로 나열한 임의의 길이의 벡터이기 때문에 제곱이 아니라 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7B%5Chat%7Bu%7D%7D_g%20%5Cmathbf%7B%5Chat%7Bu%7D%7D_g%5E%5Ctop"> term을 사용한 것입니다.</p>
<p>이에 대한 이해를 바탕으로 Cluster-robust se의 meat term을 생각해보면, <strong>Cluster-robust에서는 cluster간은 독립적이고, cluster안의 관측치들은 상관관계를 가질 수 있다고 가정하기 때문에 각 cluser에 대해서</strong> <img src="https://latex.codecogs.com/png.latex?%5CPhi_i"><strong>를</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7B%5Chat%7Bu%7D%7D_g%20%5Cmathbf%7B%5Chat%7Bu%7D%7D_g%5E%5Ctop"><strong>로 각각 구한 후, 전체</strong> <img src="https://latex.codecogs.com/png.latex?%5CPhi"><strong>는 아래와 같이 block diagonal 구조로 넣어준다고 이해</strong>할 수 있습니다. (block 행렬은 행렬을 특정한 block으로 나누었을 때 대각선 이외의 모든 행렬 블록이 영행렬인 행렬을 의미하며, cluster간의 독립을 block diagonal 구조로 고려하였다고 이해하면 됩니다.)</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CPhi_%7B%5Ctext%7Bcluster%7D%7D%20=%0A%5Cbegin%7Bpmatrix%7D%0A%5CPhi_1%20&amp;%200%20&amp;%20%5Ccdots%20&amp;%200%20%5C%5C%0A0%20&amp;%20%5CPhi_2%20&amp;%20%5Ccdots%20&amp;%200%20%5C%5C%0A%5Cvdots%20&amp;%20%5Cvdots%20&amp;%20%5Cddots%20&amp;%20%5Cvdots%20%5C%5C%0A0%20&amp;%200%20&amp;%20%5Ccdots%20&amp;%20%5CPhi_G%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p>즉, 여기서 각 <img src="https://latex.codecogs.com/png.latex?%5CPhi_g%20=%20E%5B%5Cmathbf%7Bu%7D_g%20%5Cmathbf%7Bu%7D_g%5E%5Ctop%5D">는 클러스터 <img src="https://latex.codecogs.com/png.latex?g"> 내의 오차의 공분산 행렬이고, 각 cluster에 대해 잔차<img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7Bu%7D%7D_g">를 사용하여</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D_%7B%5Ctext%7Bcluster%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%5Csum_%7Bg=1%7D%5E%7BG%7D%20%5Cmathbf%7BX%7D_g%5E%5Ctop%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%5E%5Ctop%20%5Cmathbf%7BX%7D_g%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"> 와 같이 추정합니다.</p>
<p>위 Cluster-robust의 meat항에 대한 이해는 비슷하게 3장에서도 필요하기 때문에 예시를 통해 좀더 직관적으로 보여드리겠습니다. 우선 이 중앙항은 다음과 같고,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7BB%7D%20=%20%5Csum_%7Bg=1%7D%5E%7BG%7D%20%5Cmathbf%7BX%7D_g%5E%5Ctop%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%5E%5Ctop%20%5Cmathbf%7BX%7D_g%0A"></p>
<p>이는 행렬로 보면 위에서 보신 <img src="https://latex.codecogs.com/png.latex?%5CPhi">항과 같이 block diagonal 형태를 갖습니다. 3개의 cluster가 있고, 각 cluster 내 관측치 수가 2, 3, 1개라고 가정하면 각각의 <img src="https://latex.codecogs.com/png.latex?%5CPhi">는 다음과 같고,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CPhi_1%20=%20%5Cmathbf%7BX%7D_1%5E%5Ctop%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_1%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_1%5E%5Ctop%20%5Cmathbf%7BX%7D_1%20=%0A%5Cbegin%7Bpmatrix%7D%0A%5Csigma_%7B11%7D%5E2%20&amp;%20%5Csigma_%7B12%7D%20%5C%5C%0A%5Csigma_%7B12%7D%20&amp;%20%5Csigma_%7B22%7D%5E2%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CPhi_2%20=%20%5Cmathbf%7BX%7D_2%5E%5Ctop%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_2%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_2%5E%5Ctop%20%5Cmathbf%7BX%7D_2%20=%0A%5Cbegin%7Bpmatrix%7D%0A%5Csigma_%7B33%7D%5E2%20&amp;%20%5Csigma_%7B34%7D%20&amp;%20%5Csigma_%7B35%7D%20%5C%5C%0A%5Csigma_%7B34%7D%20&amp;%20%5Csigma_%7B44%7D%5E2%20&amp;%20%5Csigma_%7B45%7D%20%5C%5C%0A%5Csigma_%7B35%7D%20&amp;%20%5Csigma_%7B45%7D%20&amp;%20%5Csigma_%7B55%7D%5E2%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5CPhi_3%20=%20%5Cmathbf%7Bx%7D_6%20%5Chat%7Bu%7D_6%5E2%20%5Cmathbf%7Bx%7D_6%5E%5Ctop%0A"></p>
<p>로 표현될 수 있으며, 결국 Cluster-robust의 중앙 term은</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cmathbf%7B%5CPhi%7D%20=%0A%5Cbegin%7Bpmatrix%7D%0A%5CPhi_1%20&amp;%200%20&amp;%200%20%5C%5C%0A0%20&amp;%20%5CPhi_2%20&amp;%200%20%5C%5C%0A0%20&amp;%200%20&amp;%20%5CPhi_3%0A%5Cend%7Bpmatrix%7D%0A"> 가 될 것입니다.</p>
<p>정리하자면, <strong>HC0는</strong> <img src="https://latex.codecogs.com/png.latex?%5CPhi">가 대각행렬인 경우로, 개별 관측치의 Heteroskedasticity 만을 고려하며</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cwidehat%7B%5Ctext%7BVar%7D%7D_%7B%5Ctext%7BHC0%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7Bx%7D_i%20%5Chat%7Bu%7D_i%5E2%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
<p>Cluster-Robust 분산 추정량은 clusr별 <img src="https://latex.codecogs.com/png.latex?%5CPhi">가 block diagonal 구조로, Heteroskedasticity와 cluster 내의 상관관계를 반영합니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AE%5Cleft%5B%5Cwidehat%7B%5Ctext%7BVar%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%5Cright%5D%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cmathbf%7BX%7D%5E%5Ctop%20%5CPhi%20%5Cmathbf%7BX%7D%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%0A"></p>
</section><section id="in-glms.." class="level3"><h3 class="anchored" data-anchor-id="in-glms..">3.4. In GLMs..</h3>
<hr>
<p>이미 위(3장에서) <strong>Cluster-robust standard errors</strong>가 어떤 원리로부터 유도되며, OLS 환경(Linear Model)에서의 공식이 어떻게 생겼는지 살펴보았습니다. 또한 <strong>HC(Heteroskedasticity-Consistent) se</strong> 역시 기본 가정(등분산, 독립성 등)이 약화되었을 때도 일관된 추정을 제공하기 위해 <strong>Robust</strong>(샌드위치) 분산 추정량을 쓰게 된다는 것을 보았습니다. GLM에서도 LM과 같이 위 두 robust한 분산 추정치 식을 사용하여 Fisher information matrix의 역행렬로 분산을 추정하는 대신, 더욱 안정적으로 분산을 추정할 수 있습니다. GLM의 경우, 단순 OLS와 달리 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BW%7D%7D,%20%5Chat%7B%5Cmathbf%7BA%7D%7D"> 등 추가적인 항이 존재하고, 이 행렬들이 실제 분산 추정 과정에 반영됩니다. 이 때문에 <strong>“bread”</strong>(양쪽에 곱해지는 행렬)와 <strong>“meat”</strong>(중간에 오는 분산·잔차 구조) 부분이 LM에서의 표기와는 형태가 조금 달라집니다. 즉, 원리는 동일하되, <strong>link &amp; variance function</strong>으로 부터 비롯된 미분 항(<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BA%7D">)과 가중치 항(<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BW%7D">)이 반영되어야 한다는 점만 다릅니다. 1장에서 소개했던 HC0를 떠올리면, LM의 경우</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D_%7B%5Ctext%7BHC0%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7Bx%7D_i%20%5Chat%7Bu%7D_i%5E2%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D,%20"></p>
<p>로 식이 구성되었습니다. <strong>GLM</strong>의 경우에는 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BW%7D%7D">가 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D">와 상호작용하여 분산 추정에 들어가므로, 실제로는 다음과 같은 형태를 가집니다. (식은 패키지나 저자별 표기 차이에 따라 다소 달라질 수 있습니다). <img src="https://latex.codecogs.com/png.latex?%20%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D_%7B%5Ctext%7BHC0%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%20%5Csum_%7Bi=1%7D%5En%20%5Cmathbf%7Bx%7D_i%20%5CBigl(%5Chat%7Bu%7D_i%5E2%20%5CBigr)%20%5Cmathbf%7Bx%7D_i%5E%5Ctop%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D)%5E%7B-1%7D.%20"></p>
<p>여기서 <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bu%7D_i">는 단순 잔차가 아니라, <strong>펄슨(pearson) 잔차</strong> 등 비선형적인 GLM 설정에 맞춰 적절히 조정된 형태일 수 있습니다. 구현별로 <strong>이탈도(deviance) 잔차</strong>를 사용할 수도 있고, 핵심은 “관측치별 잔차의 크기”를 통해 이질분산성을 추정하는 것입니다. 여기서는 앞뒤의 <img src="https://latex.codecogs.com/png.latex?(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D)%5E%7B-1%7D">이 <strong>bread</strong>(빵)이고, 가운데 잔차 <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bu%7D_i%5Chat%7Bu%7D_i%5E%5Ctop">가 <strong>meat</strong>(고기) 역할을 한다고 보면 됩니다. 철학적으로 해석하면, GLM에서도 LM에서와 동일하게 <strong>HC se</strong>는 “각 관측치별 오차분산”이 서로 다르더라도 일관된 추정을 제공하기 위하는 목적이며, 식은 (1) 잔차(오차항) 부분은 그대로 <strong>meat</strong>로 넣고, (2) 정보를 제공하는 <strong>bread</strong>에는 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D">에 가중치의 의미를 가진 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BW%7D%7D"> 항을 추가하여 구성한 위 형태로 구성됩니다.</p>
<p><strong>클러스터링이 있는 데이터</strong>에 대하여, LM과 마찬가지로 <strong>GLM에서도 Cluster-robust se</strong>가 적용될 수 있습니다. 이미 섹션 3.2~3.3에서 보았듯, 클러스터 간에는 독립이지만 클러스터 내 관측치들 간에는 상관관계가 존재할 수 있으므로, <img src="https://latex.codecogs.com/png.latex?%5CPhi"> 행렬(오차의 공분산 구조)을 <strong>block diagonal</strong> 형태로 가정하고, 이를 샌드위치 가운데(meat)에 반영합니다.</p>
<p>LM에서의 일반적 식은 다음과 같았습니다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D_%7B%5Ctext%7Bcluster%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%20%5Csum_%7Bg=1%7D%5EG%20%5Cmathbf%7BX%7D_g%5E%5Ctop%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%5E%5Ctop%20%5Cmathbf%7BX%7D_g%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Cmathbf%7BX%7D)%5E%7B-1%7D.%20"></p>
<p><strong>GLM</strong>에서는 동일한 철학으로, 단순히 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D"> 대신 가중치를 고려해 <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BW%7D%7D%5E%7B1/2%7D%5Cmathbf%7BX%7D">와 같은 형태(혹은 관련 도함수 항)가 곱해지게 됩니다. 즉,</p>
<p><img src="https://latex.codecogs.com/png.latex?%20%5Cwidehat%7B%5Cmathrm%7BVar%7D%7D_%7B%5Ctext%7Bcluster%7D%7D(%5Chat%7B%5Cboldsymbol%7B%5Cbeta%7D%7D)%20=%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D)%5E%7B-1%7D%20%5Cleft(%20%5Csum_%7Bg=1%7D%5EG%20(%5Cmathbf%7BX%7D_g%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D_g%5E%7B1/2%7D)%20%5C,%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%20%5Chat%7B%5Cmathbf%7Bu%7D%7D_g%5E%5Ctop%20%5C,%20(%5Chat%7B%5Cmathbf%7BW%7D%7D_g%5E%7B1/2%7D%20%5Cmathbf%7BX%7D_g)%20%5Cright)%20(%5Cmathbf%7BX%7D%5E%5Ctop%20%5Chat%7B%5Cmathbf%7BW%7D%7D%20%5Cmathbf%7BX%7D)%5E%7B-1%7D,%20"></p>
<p>와 같은 꼴이 됩니다(마찬가지로 패키지마다 표기 방식이나 구현 세부가 약간씩 다를 수 있습니다).각 항들 또한 한 번 더 설명하자면, <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7Bu%7D%7D_g">는 클러스터 <img src="https://latex.codecogs.com/png.latex?g"> 내 잔차 벡터(pearson 또는 deviance 잔차 등).<img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D_g">는 클러스터 <img src="https://latex.codecogs.com/png.latex?g">에 해당하는 행만 추출한 <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BX%7D">의 서브 행렬. <img src="https://latex.codecogs.com/png.latex?%5Chat%7B%5Cmathbf%7BW%7D%7D">는 클러스터 (<img src="https://latex.codecogs.com/png.latex?g">)에 해당하는 마찬가지로 가중치 서브 행렬이고, 이때 1/2승을 한다는 의미는 이가 diagonal matrix이므로 이 경우에는 단순히 diagonal 성분들 각각을 루트 씌운 값입니다.<br></p>
</section></section><section id="r-예시-glm-cluster-robust-se" class="level2"><h2 class="anchored" data-anchor-id="r-예시-glm-cluster-robust-se">4. R 예시: GLM, Cluster-robust SE</h2>
<p>아래 R 코드를 복사하여 로컬 환경에서 돌려보세요. GLM모델의 분산과 cluster-robust 분산을 비교하시면서 해석하면 됩니다.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 필요한 패키지 설치 (필요시)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## install.packages("sandwich")</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## install.packages("lmtest")</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## install.packages("nlme")</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 데이터 불러오기</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(nlme)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#data(Orthodont)  # 치아 성장 데이터 (클러스터: Subject)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#Orthodont$binary &lt;- ifelse(Orthodont$distance &gt; 25, 1, 0)  # 이항 변환</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 기본 GLM (로지스틱 회귀)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#glm_fit &lt;- glm(binary ~ age + Sex, </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#               data = Orthodont, </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#               family = binomial)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#summary(glm_fit) </span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">## 클러스터-로버스트 표준오차 (Subject 기준)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(sandwich)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#library(lmtest)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#cluster_se &lt;- vcovCL(glm_fit, cluster = ~ Subject)</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#coeftest(glm_fit, vcov = cluster_se)  # 결과 출력</span></span></code></pre></div></div>
</div>
</section><section id="마무리하며" class="level2"><h2 class="anchored" data-anchor-id="마무리하며">마무리하며</h2>
<p>이번 2장에서는 1장에서 다룬 Linear Model을 outcome of single yes/no, outcome of single K-way, count 등 non-normal한 종속변수에서도 분석할 수 있도록 확장한 <strong>Generalized linear model</strong>의 기본 개념과, 실제로 패키지에서 이 GLM의 parameter를 estimate할 때 사용하는 대표적인 알고리즘인 IRLS(Fisher scoring)을 수학적으로 상당히 깊게 살펴보았습니다. 이후, HC standard errors의 clustered data 버전인 Cluster-robust standard error를 보고, GLMs에서도 이 둘을 사용할 수 있다는 것을 밝힌 뒤 그 변형된 수식을 보았습니다. 다음 3장에서는 아직 깨지 못한 가정이었던 오차항의 독립, 즉 data(observations)간의 correlation이 존재하는 경우 자체를 모델에 반영하기 위해 개발된 모델들인 GEE, GLMM에 대하여 어느 정도 살펴보고 (GLMM의 내용은 너무 길어지기 때문에 얕게 다룰 것입니다.), 모델의 분산을 robust하게 추정하기 위한 가장 general한 형태의 Sandwich estimator를 M-estimation의 개념과 함께 공부할 것입니다.</p>


</section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{seungjun2025,
  author = {Seungjun, Lee},
  title = {Exploring {Regression} {Models} for {Regression} {Analysis}
    (2): {GLM,} {Exponential} {Family,} {Link} {Function,} {IRLS(Fisher}
    Scoring), {Cluster-robust} Standard Error},
  date = {2025-02-28},
  url = {https://blog.zarathu.com/posts/2025-02-28-reg2/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-seungjun2025" class="csl-entry quarto-appendix-citeas">
Seungjun, Lee. 2025. <span>“Exploring Regression Models for Regression
Analysis (2): GLM, Exponential Family, Link Function, IRLS(Fisher
Scoring), Cluster-Robust Standard Error.”</span> February 28, 2025. <a href="https://blog.zarathu.com/posts/2025-02-28-reg2/">https://blog.zarathu.com/posts/2025-02-28-reg2/</a>.
</div></div></section></div> ]]></description>
  <category>statistics</category>
  <guid>https://blog.zarathu.com/posts/2025-02-28-reg2/</guid>
  <pubDate>Fri, 28 Feb 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-02-28-reg2/img/reg2.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>Sample Size Estimation in MRMC</title>
  <dc:creator>Junsik CHU</dc:creator>
  <link>https://blog.zarathu.com/posts/2025-02-14-Sample Size Estimation in MRMC/</link>
  <description><![CDATA[ <!-- Google tag (gtag.js) --><script async="" src="https://www.googletagmanager.com/gtag/js?id=G-L0DYYSH9KM"></script><script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-L0DYYSH9KM');
</script><section id="introduction" class="level1"><h1>Introduction</h1>
<section id="why-is-sample-size-estimation-important" class="level2"><h2 class="anchored" data-anchor-id="why-is-sample-size-estimation-important">1. Why is Sample Size Estimation Important?</h2>
<p>의료 영상 분석에서 새로운 영상 장비의 도입이 기존 방법보다 우수한지 평가하는 것은 매우 중요한 과정이다. 이를 위해 흔히 사용되는 연구 디자인이 <strong>MRMC</strong> 연구이며, 이는 여러 명의 판독자가 여러 증례를 평가하는 방식으로 진행된다. 이러한 MRMC 연구에서의 가장 큰 고민 중 하나는 “몇 명의 판독자와 몇 개의 증례가 필요한가?” 하는 문제이다.<br>
즉, 연구의 신뢰성을 확보하면서도 불필요한 비용과 시간을 줄이기 위해 <strong>표본 크기를 적절하게 설정하는 것이 매우 중요하다.</strong></p>
</section><section id="mrmc-연구에서-표본-크기-추정의-핵심-개념" class="level2"><h2 class="anchored" data-anchor-id="mrmc-연구에서-표본-크기-추정의-핵심-개념">2. MRMC 연구에서 표본 크기 추정의 핵심 개념</h2>
<p>MRMC 연구에서는 <strong>판독자</strong>와 <strong>증례</strong>를 두 가지 주요 요소로 고려해야 한다.<br>
일반적으로 판독자가 많아질수록 데이터의 변동성이 줄어들고 연구의 신뢰성이 높아지지만, 판독자의 시간과 연구 비용을 고려해야 한다. 마찬가지로 증례 수가 많을수록 통계적 검정력이 향상되지만, 데이터 수집 및 분석 비용이 증가하게 된다.</p>
<p>또한, MRMC 연구에서는 다음과 같은 요소들이 표본 크기 결정에 영향을 미친다.</p>
<ol type="1">
<li>
<strong>효과 크기(Effect Size,</strong> <img src="https://latex.codecogs.com/png.latex?d">)
<ul>
<li>비교하고자 하는 두 판독 조건 간의 차이</li>
<li>연구 설계에서 “임상적으로 의미 있는 차이”를 미리 정의하는 것이 중요</li>
</ul>
</li>
<li>
<strong>검정력(Statistical Power,</strong> <img src="https://latex.codecogs.com/png.latex?1%20-%20%5Cbeta">)
<ul>
<li>실제로 차이가 있을 때 이를 검출할 확률</li>
<li>일반적으로 80% 이상을 목표로 설정</li>
</ul>
</li>
<li>
<strong>유의수준(Significance Level,</strong> <img src="https://latex.codecogs.com/png.latex?%5Calpha">)
<ul>
<li>Type I 오류의 허용 범위</li>
<li>보통 5%로 설정</li>
</ul>
</li>
<li>
<strong>분산 성분(Variance Components)</strong>
<ul>
<li>판독자 간 변동성, 증례 간 변동성, 판독자와 증례 간 상호작용 등의 변동 요인을 반영</li>
</ul>
</li>
</ol></section><section id="mrmc-연구에서의-성능-평가" class="level2"><h2 class="anchored" data-anchor-id="mrmc-연구에서의-성능-평가">3. MRMC 연구에서의 성능 평가</h2>
<p>의료 영상 분석에서는 <strong>병변의 유무를 파악하는 것이 중요</strong>하다.<br>
그 성능을 평가하는 대표적인 방법으로는 <strong>ROC(Receiver Operating Characteristic) 분석</strong>이 있는데, 이는 병변의 단순 유무를 파악하는 데 사용된다.<br>
그러나 영상 판독 연구에서는 단순히 병변의 유무뿐만 아니라 <strong>병변의 위치(Localization)</strong>까지 고려하는 것이 중요하다.<br>
이때 <strong>FROC(Free-Response ROC) 분석</strong>이 사용된다.</p>
<p>FROC 분석을 활용한 MRMC 연구에서는 판독자가 병변의 위치까지도 설정하도록 하여,<br><strong>병변 탐지(Detection)</strong>뿐만 아니라 <strong>위치 정확도(Localization Accuracy)</strong>도 평가된다.<br>
권장되는 평가 지표로는 <strong>가중 AFROC(Weighted AFROC, wAFROC)</strong> 가 있으며,<br>
모든 병변은 동일한 가중치를 갖는 것이 일반적이다(특정 임상적인 이유가 있다면 병변마다 다른 가중치를 부여할 수 있음).</p>
<hr></section></section><section id="statistical-framework-for-sample-size-estimation" class="level1"><h1>Statistical Framework for Sample Size Estimation</h1>
<section id="hypothesis-testing-in-mrmc-studies" class="level2"><h2 class="anchored" data-anchor-id="hypothesis-testing-in-mrmc-studies">1. Hypothesis Testing in MRMC Studies</h2>
<p>MRMC 연구에서 <strong>새로운 영상 기술(또는 판독 조건)이 기존 방법보다 우수한지 검증</strong>하려면 통계적 검정을 수행해야 한다.<br>
이를 위해 귀무가설(<img src="https://latex.codecogs.com/png.latex?H_0">)과 대립가설(<img src="https://latex.codecogs.com/png.latex?H_1">)을 설정하고, 효과 크기(Effect Size)를 기반으로 표본 크기를 결정한다.</p>
<ul>
<li>
<strong>귀무가설(</strong><img src="https://latex.codecogs.com/png.latex?H_0">): 두 판독 조건(예: 기존 방법 vs.&nbsp;새로운 방법)의 성능 차이가 없다.
<ul>
<li>즉, <img src="https://latex.codecogs.com/png.latex?%5Ctheta_1%20=%20%5Ctheta_2"> (ROC-AUC 또는 wAFROC-AUC 값이 동일)</li>
</ul>
</li>
<li>
<strong>대립가설(</strong><img src="https://latex.codecogs.com/png.latex?H_1">): 두 판독 조건의 성능 차이가 있다.
<ul>
<li>즉, <img src="https://latex.codecogs.com/png.latex?%5Ctheta_1%20%5Cneq%20%5Ctheta_2">
</li>
</ul>
</li>
</ul>
<hr></section><section id="observed-vs.-anticipated-effect-size" class="level2"><h2 class="anchored" data-anchor-id="observed-vs.-anticipated-effect-size">2. Observed vs.&nbsp;Anticipated Effect-Size</h2>
<p>표본 크기를 결정하려면 <strong>두 판독 조건 간의 차이를 수량화한 효과 크기(Effect Size,</strong> <img src="https://latex.codecogs.com/png.latex?d">)를 추정해야 한다.<br>
이때 <strong>관측된 효과 크기(Observed Effect-Size)와 예상 효과 크기(Anticipated Effect-Size)</strong>를 구분해야 한다.</p>
<p>✔ <strong>관측된 효과 크기(</strong><img src="https://latex.codecogs.com/png.latex?d_%7B%5Ctext%7Bobs%7D%7D">):<br>
- 파일럿 연구(Pilot Study)에서 얻은 효과 크기의 추정값<br>
- 작은 표본 크기로 인해 오차가 포함될 가능성이 있음<br>
✔ <strong>예상 효과 크기(</strong><img src="https://latex.codecogs.com/png.latex?d_%7B%5Ctext%7Bant%7D%7D">):<br>
- 본 연구(Pivotal Study)의 표본 크기를 결정하기 위해 설정하는 값<br>
- <strong>신뢰구간(Confidence Interval, CI)을 고려하여 설정해야 함</strong></p>
<p>예상 효과 크기를 설정할 때 다음과 같은 접근법이 가능하다.</p>
<ol type="1">
<li>
<strong>신뢰구간의 하한을 사용 (보수적 접근법)</strong>
<ul>
<li>Type II 오류를 줄여 검정력을 확보 가능</li>
<li>그러나 불필요하게 많은 판독자 및 증례가 필요할 수 있음</li>
</ul>
</li>
<li>
<strong>신뢰구간의 중앙값을 사용 (균형적 접근법)</strong>
<ul>
<li>너무 보수적이지 않으면서도 무리한 연구 설계를 방지 가능</li>
</ul>
</li>
<li>
<strong>관측된 효과 크기 그대로 사용 (위험 부담 있음)</strong>
<ul>
<li>파일럿 연구 표본이 작으면 변동성이 크므로 추천되지 않음</li>
</ul>
</li>
</ol>
<hr></section><section id="variance-components-and-their-impact-on-power" class="level2"><h2 class="anchored" data-anchor-id="variance-components-and-their-impact-on-power">3. Variance Components and Their Impact on Power</h2>
<p>MRMC 연구에서 표본 크기를 결정할 때, 효과 크기(Effect Size)뿐만 아니라 <strong>분산 성분(Variance Components)이 통계적 검정력(Statistical Power)에 미치는 영향</strong>을 고려해야 한다.<br>
분산 성분은 <strong>판독자 간 변동성(Reader Variability), 증례 간 변동성(Case Variability), 판독자-처치 간 변동성(Treatment-Reader Interaction Variability)</strong> 등을 반영하며, 연구 설계에서 중요한 요소이다.</p>
<section id="통계적-검정력statistical-power과-분산-성분의-관계" class="level3"><h3 class="anchored" data-anchor-id="통계적-검정력statistical-power과-분산-성분의-관계"><strong>3.1. 통계적 검정력(Statistical Power)과 분산 성분의 관계</strong></h3>
<p>MRMC 연구에서 검정력은 효과 크기(<img src="https://latex.codecogs.com/png.latex?d">)와 분산 성분의 함수로 결정된다. 통계적 검정력은 <strong>비중심 모수(Non-centrality Parameter,</strong> <img src="https://latex.codecogs.com/png.latex?%5Clambda">) 에 의해 결정되며, 다음과 같은 조건에서 검정력이 증가한다.</p>
<p>검정력은 다음과 같은 확률로 정의된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BPower%7D%20=%20P%5Cleft(F%20%3E%20F_%7B%5Calpha,%20%5Ctext%7Bndf%7D,%20%5Ctext%7Bddf%7D%7D%20%5Cmid%20%5Clambda%20%5Cright)%0A"></p>
<p>여기서,<br>
- <img src="https://latex.codecogs.com/png.latex?F"> : 검정 통계량 (F-statistic)<br>
- <img src="https://latex.codecogs.com/png.latex?F_%7B%5Calpha,%20%5Ctext%7Bndf%7D,%20%5Ctext%7Bddf%7D%7D"> : <strong>유의수준</strong> <img src="https://latex.codecogs.com/png.latex?%5Calpha">에서의 F-분포 임계값 (분자 자유도 <img src="https://latex.codecogs.com/png.latex?%5Ctext%7Bndf%7D">, 분모 자유도 <img src="https://latex.codecogs.com/png.latex?%5Ctext%7Bddf%7D">)<br>
- <img src="https://latex.codecogs.com/png.latex?%5Clambda"> : <strong>비중심 모수(Non-centrality Parameter)</strong></p>
<p>비중심 모수 <img src="https://latex.codecogs.com/png.latex?%5Clambda"> 는 다음과 같이 정의된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Clambda%20=%20%5Cfrac%7BJ%20K%20d%5E2%7D%7B%5Csigma%5E2_%7B%5Ctext%7Btotal%7D%7D%7D%0A"></p>
<p>여기서,<br>
- <img src="https://latex.codecogs.com/png.latex?J"> : 판독자 수(Readers)<br>
- <img src="https://latex.codecogs.com/png.latex?K"> : 증례 수(Cases)<br>
- <img src="https://latex.codecogs.com/png.latex?d"> : 효과 크기(Effect Size)<br>
- <img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_%7B%5Ctext%7Btotal%7D%7D"> : 총 분산 성분 (전체 변동성)</p>
</section><section id="분산-성분이-검정력에-미치는-영향" class="level3"><h3 class="anchored" data-anchor-id="분산-성분이-검정력에-미치는-영향"><strong>3.2. 분산 성분이 검정력에 미치는 영향</strong></h3>
<section id="분자가-클-경우-검정력-증가" class="level4"><h4 class="anchored" data-anchor-id="분자가-클-경우-검정력-증가"><strong>(1) 분자가 클 경우 → 검정력 증가</strong></h4>
<ol type="1">
<li>
<strong>예상 효과 크기(Anticipated Effect-Size,</strong> <img src="https://latex.codecogs.com/png.latex?d_%7B%5Ctext%7Bant%7D%7D">)가 클 경우
<ul>
<li>
<strong>효과 크기가 클수록 귀무가설(</strong><img src="https://latex.codecogs.com/png.latex?H_0">)을 기각할 가능성이 높아진다.</li>
</ul>
</li>
<li>판독자 수(<img src="https://latex.codecogs.com/png.latex?J">) 또는 증례 수(<img src="https://latex.codecogs.com/png.latex?K">)가 클 경우
<ul>
<li>판독자(<img src="https://latex.codecogs.com/png.latex?J">) 또는 증례(<img src="https://latex.codecogs.com/png.latex?K">)가 많을수록 <strong>추정값의 변동성이 감소</strong>하여 통계적 검정력이 증가한다.</li>
<li>이는 <strong>큰 표본 크기가 귀무가설 기각 확률을 증가시키는 일반적인 원리</strong>와 동일하다.</li>
</ul>
</li>
</ol></section><section id="분모가-작을-경우-검정력-증가" class="level4"><h4 class="anchored" data-anchor-id="분모가-작을-경우-검정력-증가"><strong>(2) 분모가 작을 경우 → 검정력 증가</strong></h4>
<p>통계적 검정력의 분모는 다음과 같이 구성된다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Csigma%5E2_%7B%5Cepsilon%7D%20+%20%5Csigma%5E2_%7B%5Ctau%20RC%7D%20+%20K%5Csigma%5E2_%7B%5Ctau%20R%7D%20+%20J%5Csigma%5E2_%7B%5Ctau%20C%7D%0A"></p>
<p>이때, <strong>각 분산 성분이 작아질수록 검정력이 증가한다.</strong></p>
<ol type="1">
<li>
<p><strong>잔차 변동성(</strong><img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_%7B%5Cepsilon%7D%20+%20%5Csigma%5E2_%7B%5Ctau%20RC%7D">) 감소 → 검정력 증가</p>
<ul>
<li>이 두 항은 Jackknife Pseudovalues의 잔차 변동성을 나타낸다.</li>
<li>잔차 변동성이 작을수록 <strong>비중심 모수(Non-centrality Parameter,</strong> <img src="https://latex.codecogs.com/png.latex?%5Clambda">)가 증가하여 검정력이 증가한다.</li>
</ul>
</li>
<li>
<p><strong>처치-판독자 변동성(</strong><img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_%7B%5Ctau%20R%7D">) 감소 → 검정력 증가<br>
※ 처치 : 비교하고자하는 판독 조건을 의미;의료 영상 분석의 예시에서는 기존 방식과 AI 보조 영상 판독가 될 수 있음</p>
<ul>
<li>판독자 자체의 변동성(<img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_R">)은 모든 처치 조건에서 동일한 영향을 미치므로 검정력에 영향을 주지 않는다.</li>
<li>처치-판독자(Treatment-Reader) 분산 성분(<img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_%7B%5Ctau%20R%7D">)은 <strong>노이즈로 작용</strong>하여 효과 크기 추정을 방해하고 검정력을 감소시킨다.</li>
<li>따라서 <strong>이 값이 작을수록 표본 크기 추정의 정확도가 높아진다.</strong>
</li>
</ul>
</li>
<li>
<p><strong>처치-증례 변동성(</strong><img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_%7B%5Ctau%20C%7D">) 감소 → 검정력 증가</p>
<ul>
<li>증례 자체의 변동성(<img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_C">)은 모든 처치 조건에서 동일한 영향을 미치므로 검정력에 영향을 주지 않는다.</li>
<li>그러나 <strong>처치-증례 변동성(</strong><img src="https://latex.codecogs.com/png.latex?%5Csigma%5E2_%7B%5Ctau%20C%7D">)은 효과 크기 추정치에 <strong>노이즈</strong>를 추가하여 검정력을 감소시킨다.</li>
<li>이 항은 <strong>판독자 수(</strong><img src="https://latex.codecogs.com/png.latex?J">)와 곱해지므로, 보통의 <img src="https://latex.codecogs.com/png.latex?J%20%5Cll%20K">인 연구에서는 그 영향이 크지 않다.</li>
</ul>
</li>
</ol>
<p>📌 **즉, MRMC 연구 설계에서 표본 크기를 결정할 때, 단순히 숫자를 늘리는 것이 아니라, 이러한 분산 성분을 고려해야 한다.</p>
<hr></section></section></section></section><section id="sample-size-estimation-for-roc-studies" class="level1"><h1>Sample Size Estimation for ROC Studies</h1>
<p>MRMC 연구에서 <strong>ROC(Receiver Operating Characteristic) 분석</strong>을 기반으로 한 <strong>샘플 크기 추정</strong>은 연구 설계에서 필수적인 단계이다.<br>
특히, 필요한 판독자 수(<img src="https://latex.codecogs.com/png.latex?J">)와 증례 수(<img src="https://latex.codecogs.com/png.latex?K">) 조합별 검정력 변화를 분석하는 것이 중요하다.</p>
<section id="using-pilot-studies" class="level2"><h2 class="anchored" data-anchor-id="using-pilot-studies"><strong>1. Using Pilot Studies</strong></h2>
<p>파일럿 연구(Pilot Study)는 본 연구에서 필요한 효과 크기(<img src="https://latex.codecogs.com/png.latex?d_%7B%5Ctext%7Bobs%7D%7D">) 및 분산 성분을 추정하는 역할을 한다. - 파일럿 연구에서 얻은 정보는 <code>RJafroc</code> 패키지의 <code>SsSampleSizeKGiven()</code> 함수를 활용해 표본 크기를 결정하는 데 사용된다.</p>
</section><section id="sample-size-estimation-using-rjafroc" class="level2"><h2 class="anchored" data-anchor-id="sample-size-estimation-using-rjafroc"><strong>2. Sample Size Estimation Using RJafroc</strong></h2>
<section id="함수-개요" class="level3"><h3 class="anchored" data-anchor-id="함수-개요"><strong>2.1. 함수 개요</strong></h3>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># SsSampleSizeKGivenJ()</span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># SsSampleSizeKGivenJ(dataset, FOM, J, effectSize, method, analysisOption)</span></span></code></pre></div></div>
</div>
<ul>
<li>dataset : Pilot Data</li>
<li>FOM : 평가 지표 (“ROC-AUC” in ROC Studies)</li>
<li>J : 판독자 수</li>
<li>K : 증례 수</li>
<li>effectSize : 효과 크기 (파일럿 연구에서 추정)</li>
<li>method : 분석 방법 (“DBM” 또는 “OR”)</li>
<li>analysisOption : 분산 분석 Option (“RRRC”, “FRRC”, “RRFC”)</li>
</ul></section><section id="실제-예제" class="level3"><h3 class="anchored" data-anchor-id="실제-예제"><strong>2.2. 실제 예제</strong></h3>
<p>일반적인 연구 설계에선 판독자수는 제한된 자원으로 인해 고정되거나, 대략적인 좁은 범위에서 결정되는 경우가 많다.</p>
<p>이에 특정 판독자 수에서 검정력을 만족하는 최소한의 증례 수를 찾는 것이 일반적이고, RJafroc의 SsSampleSizeKGivenJ 함수도 그러하다.</p>
<p>본 예시 코드에서는 판독자수가 6~13명 정도의 범위일때, 검정력이 80% 이상이 되는 <strong>최소한의</strong> <img src="https://latex.codecogs.com/png.latex?K"> 값을 찾는다.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2" style="background: #f1f3f5;"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;"><a href="https://dpc10ster.github.io/RJafroc/">RJafroc</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pilot_data</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">dataset02</span>  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 예제 데이터셋 사용</span></span>
<span></span>
<span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">target_power</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.8</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 목표 검정력 </span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">J_vals</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">6</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">13</span> <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 판독자 수 범위 지정 </span></span>
<span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">optimal_K_results</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>J <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/integer.html">integer</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, K <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/integer.html">integer</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>, Power <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/numeric.html">numeric</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span>  </span>
<span></span>
<span><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># 목표 검정력을 만족하는 최소 K(증례수) 도출 </span></span>
<span><span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">J</span> <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">in</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">J_vals</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">{</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ret</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">SsSampleSizeKGivenJ</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span></span>
<span>    dataset <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">pilot_data</span>,</span>
<span>    FOM <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Wilcoxon"</span>,</span>
<span>    J <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">J</span>,</span>
<span>    analysisOption <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"RRRC"</span>,</span>
<span>    alpha <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span>,</span>
<span>    desiredPower <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">target_power</span></span>
<span>  <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span>  <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">optimal_K_results</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/cbind.html">rbind</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">optimal_K_results</span>, </span>
<span>                             <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span>J <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">J</span>, K <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ret</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">K</span>, Power <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://rdrr.io/r/base/Round.html">signif</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">ret</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">powerRRRC</span>, <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span>
<span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span></span>
<span></span>
<span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rmarkdown</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;"><a href="https://pkgs.rstudio.com/rmarkdown/reference/paged_table.html">paged_table</a></span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">(</span><span class="va" style="color: #111111;
background-color: null;
font-style: inherit;">optimal_K_results</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">)</span></span></code></pre></div></div>
<div data-pagedtable="false">
  <script data-pagedtable-source="" type="application/json">
{"columns":[{"label":["J"],"name":[1],"type":["int"],"align":["right"]},{"label":["K"],"name":[2],"type":["dbl"],"align":["right"]},{"label":["Power"],"name":[3],"type":["dbl"],"align":["right"]}],"data":[{"1":"6","2":"251","3":"0.801"},{"1":"7","2":"211","3":"0.801"},{"1":"8","2":"188","3":"0.801"},{"1":"9","2":"173","3":"0.801"},{"1":"10","2":"163","3":"0.802"},{"1":"11","2":"155","3":"0.801"},{"1":"12","2":"149","3":"0.802"},{"1":"13","2":"144","3":"0.801"}],"options":{"columns":{"min":{},"max":[10]},"rows":{"min":[10],"max":[10]},"pages":{}}}
  </script>
</div>
<ul>
<li>위 코드를 실행하면 판독자가 많아짐에 따라 같은 효과 크기에서도 더 작은 증례수로 목표 검정력을 확보할 수 있음을 확인 가능하다.</li>
<li>SsSampleSizeKGivenJ() 함수를 이용하면 파일럿 연구에서 추정된 효과 크기와 목표 검정력을 바탕으로 MRMC 연구의 판독자수에 따른 적절한 증례 수를 합리적으로 결정할 수 있게 된다.</li>
</ul>
<hr></section></section></section><section id="sample-size-estimation-for-froc-studies" class="level1"><h1>Sample Size Estimation for FROC Studies</h1>
<p>FROC 연구에서는 ROC 연구에서 활용했던 effect size를 그대로 사용할 수 없다. 따라서 ROC 연구에서의 effect size를 변환해주는 과정이 필요하다.</p>
<section id="froc와-roc-연구의-차이점" class="level2"><h2 class="anchored" data-anchor-id="froc와-roc-연구의-차이점">1. FROC와 ROC 연구의 차이점</h2>
<p>FROC 연구에서 효과 크기 변환이 필요한 이유를 이해하려면, <strong>ROC와 wAFROC의 차이</strong>를 먼저 살펴보아야 한다.</p>
<section id="roc-연구의-특징" class="level3"><h3 class="anchored" data-anchor-id="roc-연구의-특징">ROC 연구의 특징</h3>
<ul>
<li>
<strong>판독자가 병변이 있는지(Yes/No)만 결정</strong> → 위치 정보는 고려되지 않음</li>
<li>
<strong>성능 평가 지표:</strong> ROC-AUC (0.5 ~ 1)
<ul>
<li>무작위 성능 ≈ <strong>0.5</strong> (랜덤으로 판별할 경우)</li>
<li>완벽한 성능 ≈ <strong>1</strong>
</li>
</ul>
</li>
</ul></section><section id="froc-연구의-특징" class="level3"><h3 class="anchored" data-anchor-id="froc-연구의-특징">FROC 연구의 특징</h3>
<ul>
<li>
<strong>판독자가 병변이 있는 위치까지 특정해야 함</strong> → 탐지뿐만 아니라 위치 정확도(Localization Accuracy)도 고려됨</li>
<li>
<strong>성능 평가 지표:</strong> wAFROC-AUC (0 ~ 1)
<ul>
<li>무작위 성능 ≈ <strong>0</strong> (무작위로 위치를 선택하면 병변을 정확히 찾을 확률이 0에 가까움)</li>
<li>완벽한 성능 ≈ <strong>1</strong>
</li>
</ul>
</li>
</ul>
<p>✔ <strong>ROC-AUC는 0.5 ~ 1 범위에서 변화하는 반면, wAFROC-AUC는 0 ~ 1 범위에서 변화한다.</strong><br>
✔ <strong>따라서 판독 능력의 차이가 비슷하더라도, 수치적인 AUC 차이는 서로 다르게 나타날 수 있다.</strong><br>
✔ <strong>따라서 ROC 연구에서 정의한 효과 크기를 FROC 연구에서 적용하려면 변환 과정이 필요하다.</strong></p>
<hr></section></section><section id="변환을-위한-rsmradiological-search-model-적용" class="level2"><h2 class="anchored" data-anchor-id="변환을-위한-rsmradiological-search-model-적용">2. 변환을 위한 RSM(Radiological Search Model) 적용</h2>
<p>FROC 연구에서 <strong>ROC 효과 크기를 wAFROC 효과 크기로 변환하려면</strong><br><strong>Radiological Search Model (RSM)</strong>을 활용해야 한다.</p>
<section id="rsm-모델의-핵심-개념" class="level3"><h3 class="anchored" data-anchor-id="rsm-모델의-핵심-개념">RSM 모델의 핵심 개념</h3>
<p>RSM 모델은 판독자의 <strong>탐색(Search)과 의사결정(Decision Making) 과정</strong>을 모델링하며,<br>
이를 위해 3가지 주요 매개변수를 사용한다.</p>
<ul>
<li>
<strong>μ (mu): 병변과 비병변 간의 신호 대비 (Signal-to-Noise Ratio, SNR)</strong>
<ul>
<li><strong>ROC에서 병변을 감지하는 능력(Detection Ability)을 나타냄</strong></li>
<li>값이 클수록 병변과 비병변을 더 잘 구별할 수 있음</li>
</ul>
</li>
<li>
<strong>λ (lambda): 비병변을 오탐(False Positive)할 가능성</strong>
<ul>
<li>값이 클수록 <strong>판독자가 비병변(non-lesion)을 병변으로 잘못 판단할 확률이 증가</strong>
</li>
</ul>
</li>
<li>
<strong>ν (nu): 실제 병변을 정확히 찾는 확률 (Localization Accuracy)</strong>
<ul>
<li>값이 클수록 <strong>판독자가 병변을 정확한 위치에 마킹할 가능성이 증가</strong>
</li>
</ul>
</li>
</ul>
<hr></section></section><section id="변환-과정" class="level2"><h2 class="anchored" data-anchor-id="변환-과정">3. 변환 과정</h2>
<section id="변환-계수scaling-factor-계산" class="level3"><h3 class="anchored" data-anchor-id="변환-계수scaling-factor-계산">변환 계수(Scaling Factor) 계산</h3>
<p>변환 계수는 ROC 효과 크기(<img src="https://latex.codecogs.com/png.latex?%5CDelta">ROC-AUC)를 FROC 효과 크기(<img src="https://latex.codecogs.com/png.latex?%5CDelta">wAFROC-AUC)로 변환하기 위한 비율이며, 다음 절차를 통해 도출된다.</p>
<hr>
<section id="파일럿-데이터에서-rsm-모델-적합fitting-및-auc-계산" class="level4"><h4 class="anchored" data-anchor-id="파일럿-데이터에서-rsm-모델-적합fitting-및-auc-계산">1. 파일럿 데이터에서 RSM 모델 적합(fitting) 및 AUC 계산</h4>
<p>✔ 파일럿 데이터에서 <strong>RSM 모델을 적합(fit)하여</strong> ROC-AUC 및 wAFROC-AUC 값을 도출한다.<br>
✔ 이를 통해 현재의 <img src="https://latex.codecogs.com/png.latex?%5Cmu">, <img src="https://latex.codecogs.com/png.latex?%5Clambda">, <img src="https://latex.codecogs.com/png.latex?%5Cnu"> 값을 얻는다.</p>
<hr></section><section id="mu를-단계적으로-증가시키면서-roc-auc와-wafroc-auc-변화를-측정" class="level4"><h4 class="anchored" data-anchor-id="mu를-단계적으로-증가시키면서-roc-auc와-wafroc-auc-변화를-측정">2. <img src="https://latex.codecogs.com/png.latex?%5Cmu">를 단계적으로 증가시키면서 ROC-AUC와 wAFROC-AUC 변화를 측정</h4>
<p>✔ <strong>모델의 탐지 능력이 향상된다고 가정하고</strong> <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 값을 일정한 간격(예: <img src="https://latex.codecogs.com/png.latex?%5CDelta%20%5Cmu%20=%200.01">)으로 증가시킨다.<br>
✔ <img src="https://latex.codecogs.com/png.latex?%5Cmu">가 증가함에 따라 ROC-AUC와 wAFROC-AUC도 증가하게 된다.<br>
✔ 그러나 <img src="https://latex.codecogs.com/png.latex?%5Clambda">와 <img src="https://latex.codecogs.com/png.latex?%5Cnu">는 <img src="https://latex.codecogs.com/png.latex?%5Cmu">와 상관성이 있기 때문에, <img src="https://latex.codecogs.com/png.latex?%5Cmu">만 단순 증가시키면 변환이 정확하지 않다.<br>
✔ 이를 해결하기 위해, <strong>내재적 파라미터 변환을 적용하여</strong> <img src="https://latex.codecogs.com/png.latex?%5Clambda">와 <img src="https://latex.codecogs.com/png.latex?%5Cnu">도 함께 변화하도록 조정한다.</p>
<hr></section><section id="내재적-파라미터-변환-util2intrinsic-사용" class="level4"><h4 class="anchored" data-anchor-id="내재적-파라미터-변환-util2intrinsic-사용">3. 내재적 파라미터 변환 (<code>Util2Intrinsic()</code> 사용)</h4>
<p>✔ <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 값이 변화하면 <img src="https://latex.codecogs.com/png.latex?%5Clambda">와 <img src="https://latex.codecogs.com/png.latex?%5Cnu">도 함께 조정되어야 하는데, 직접적으로 조정하기 어렵다.<br>
✔ 따라서 <strong>물리적(Physical) 파라미터 (</strong><img src="https://latex.codecogs.com/png.latex?%5Cmu,%20%5Clambda,%20%5Cnu">)를 내재적(Intrinsic) 파라미터 (<img src="https://latex.codecogs.com/png.latex?%5Clambda_i,%20%5Cnu_i">)로 변환한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Clambda_i%20=%20%5Clambda%20%5Ctimes%20%5Cmu%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cnu_i%20=%20-%5Clog(1%20-%20%5Cnu)%20/%20%5Cmu%0A"></p>
<p>✔ 변환 후, <img src="https://latex.codecogs.com/png.latex?%5Cmu">를 증가시키면서도 <img src="https://latex.codecogs.com/png.latex?%5Clambda">와 <img src="https://latex.codecogs.com/png.latex?%5Cnu">를 올바르게 유지할 수 있는 상태가 된다.</p>
<hr></section><section id="각-단계에서-roc-auc-변화량-deltaroc-auc과-wafroc-auc-변화량-deltawafroc-auc-비교" class="level4"><h4 class="anchored" data-anchor-id="각-단계에서-roc-auc-변화량-deltaroc-auc과-wafroc-auc-변화량-deltawafroc-auc-비교">4. 각 단계에서 ROC-AUC 변화량 (<img src="https://latex.codecogs.com/png.latex?%5CDelta">ROC-AUC)과 wAFROC-AUC 변화량 (<img src="https://latex.codecogs.com/png.latex?%5CDelta">wAFROC-AUC) 비교</h4>
<p>✔ <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 값을 증가시킨 후, 해당하는 새로운 ROC-AUC와 wAFROC-AUC를 계산한다.<br>
✔ 그러나 이 상태에서는 여전히 기존 <img src="https://latex.codecogs.com/png.latex?%5Clambda">와 <img src="https://latex.codecogs.com/png.latex?%5Cnu">를 사용하고 있으므로, 이를 새로운 <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 값에 맞게 변환해야 한다.</p>
<section id="내재적-물리적-변환-util2physical-사용" class="level5"><h5 class="anchored" data-anchor-id="내재적-물리적-변환-util2physical-사용">내재적 → 물리적 변환 (<code>Util2Physical()</code> 사용)</h5>
<p>✔ 증가된 <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 값(<img src="https://latex.codecogs.com/png.latex?%5Cmu_%7B%5Ctext%7Bnew%7D%7D">)에 대해 <strong>내재적 파라미터를 다시 물리적 파라미터로 변환</strong>하여 <img src="https://latex.codecogs.com/png.latex?%5Clambda">와 <img src="https://latex.codecogs.com/png.latex?%5Cnu">를 조정한다.</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Clambda%20=%20%5Cfrac%7B%5Clambda_i%7D%7B%5Cmu_%7B%5Ctext%7Bnew%7D%7D%7D%0A"></p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cnu%20=%201%20-%20%5Cexp(-%5Cnu_i%20%5Ctimes%20%5Cmu_%7B%5Ctext%7Bnew%7D%7D)%0A"></p>
<p>✔ 즉, 새로운 <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 값에서 RSM 모델을 다시 적합하여, 해당하는 ROC-AUC 및 wAFROC-AUC를 구한다.<br>
✔ 이를 반복 수행하여 <img src="https://latex.codecogs.com/png.latex?%5CDelta">ROC-AUC과 <img src="https://latex.codecogs.com/png.latex?%5CDelta">wAFROC-AUC을 측정한다.</p>
<hr></section></section><section id="선형-회귀를-통해-변환-계수scaling-factor-결정" class="level4"><h4 class="anchored" data-anchor-id="선형-회귀를-통해-변환-계수scaling-factor-결정">5. 선형 회귀를 통해 변환 계수(Scaling Factor) 결정</h4>
<p>✔ 여러 번의 <img src="https://latex.codecogs.com/png.latex?%5Cmu"> 증가 단계에 대해 <img src="https://latex.codecogs.com/png.latex?%5CDelta">ROC-AUC(독립 변수)와 <img src="https://latex.codecogs.com/png.latex?%5CDelta">wAFROC-AUC(종속 변수) 사이의 관계를 분석한다.<br>
✔ 이를 선형 회귀(linear regression)를 사용하여 분석하면 <strong>변환 계수(Scaling Factor)</strong>를 구할 수 있다.</p>
<p>✔ 선형 회귀 분석에서 <img src="https://latex.codecogs.com/png.latex?%5CDelta">ROC-AUC를 독립 변수, <img src="https://latex.codecogs.com/png.latex?%5CDelta">wAFROC-AUC를 종속 변수로 설정하면 기울기(Slope)가 변환 계수가 된다.</p>
<p>✔ <strong>위 과정을 통해 ROC 연구에서의 효과 크기를 FROC 연구에서 사용할 수 있도록 변환할 수 있다.</strong></p>
<hr></section></section></section><section id="변환-후-표본-크기-추정" class="level2"><h2 class="anchored" data-anchor-id="변환-후-표본-크기-추정">4. 변환 후 표본 크기 추정</h2>
<p>변환 계수를 적용하고 나면, ROC 연구와 동일한 방법으로 FROC 연구에서 필요한 표본 크기를 계산할 수 있다.</p>
<ol type="1">
<li><p><strong>ROC 연구에서 정의한 효과 크기(ΔROC-AUC)를 변환하여 wAFROC 효과 크기(ΔwAFROC-AUC)로 변환</strong></p></li>
<li><p><strong>변환된 wAFROC 효과 크기를 기반으로, SsSampleSizeKGivenJ() 함수를 이용하여 샘플 크기 계산</strong></p></li>
</ol>
<hr></section></section><section id="conclusion" class="level1"><h1>Conclusion</h1>
<p>이번 글에서는 MRMC 연구에서 표본 크기 추정의 중요성과 그 방법론을 살펴보았다.</p>
<p>파일럿 데이터를 통해 추정한 효과 크기와 분산 성분을 바탕으로, RJafroc 패키지의 기능을 활용하여 적절한 증례 수를 산출하는 과정을 살펴보았으며,<br>
ROC 연구에서 도출된 효과 크기를 FROC 연구에 맞게 변환하는 RSM 모델 기반의 접근법도 소개하였다.</p>
<p>결론적으로, MRMC 연구에서 체계적인 표본 크기 추정은 연구의 신뢰성을 높이고, 임상 환경에 맞는 효율적인 연구 설계를 가능하게 한다.</p>
<section id="reference" class="level2"><h2 class="anchored" data-anchor-id="reference">Reference</h2>
<p>https://cran.r-project.org/web/packages/RJafroc/RJafroc.pdf<br>
https://dpc10ster.github.io/RJafrocBook/</p>


</section></section><div id="quarto-appendix" class="default"><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre class="sourceCode code-with-copy quarto-appendix-bibtex"><code class="sourceCode bibtex">@online{chu2025,
  author = {CHU, Junsik},
  title = {Sample {Size} {Estimation} in {MRMC}},
  date = {2025-02-14},
  url = {https://blog.zarathu.com/posts/2025-02-14-Sample Size Estimation in MRMC/},
  langid = {en}
}
</code></pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-chu2025" class="csl-entry quarto-appendix-citeas">
CHU, Junsik. 2025. <span>“Sample Size Estimation in MRMC.”</span>
February 14, 2025. <a href="https://blog.zarathu.com/posts/2025-02-14-Sample Size Estimation in MRMC/">https://blog.zarathu.com/posts/2025-02-14-Sample
Size Estimation in MRMC/</a>.
</div></div></section></div> ]]></description>
  <category>R</category>
  <guid>https://blog.zarathu.com/posts/2025-02-14-Sample Size Estimation in MRMC/</guid>
  <pubDate>Fri, 14 Feb 2025 00:00:00 GMT</pubDate>
  <media:content url="https://blog.zarathu.com/posts/2025-02-14-Sample Size Estimation in MRMC/img/RJafroc.png" medium="image" type="image/png" height="72" width="144"/>
</item>
</channel>
</rss>
