시간과 공간 그리고 진실의 방

바야흐로 문제를 정의하고 나니, 사태는 약 2월달 중후반으로 거슬러 올라가자면

해당 사태는 예전의 예전으로 거슬러 올라가 사용자들로 부터 알게된 것이었으니...

아니 개발자님 앱 알림이 안오는데여?

문제는 이 유저들은 어디까지 선량한 사람들인 것인지 앱 알림이 오지 않으면 문의를 하거나 고객센터에 뭘 남겼어야 한다는 것인데, 아무것도 알려오지 않고 그저 자유게시판에 글만 써오고 있었다는 사실이었다.

그래서 왜때문인고? 하면서 추적을 시작하는 가운데 들어가보니

 Google Cloud의 Firebase Messaging 이 작동할 수 없는 상태로 되어버린 거시고?

그 상태를 추적하다 보니 프로젝트 콘솔 자체가 중지되어버린 것이였다.

내용인 즉슨

님 프로젝트 API키가 노출되었거나 하이재킹 이용되고 있는데 왜 점검안함? 정신안차림?

ㅓ... 오만가지 생각이 지나가면서 나는 어디 부분에서 노출되었나 생각했고, 일단 Claim을 하면서 다음과 같이 문서를 보냈다.

Dear Google Cloud Trust & Safety Team,

Thank you for your response.

Unfortunately, because the project remains suspended, I have been unable to access the Google Cloud Console to review logs, inspect resources, or identify the exact activity that triggered the violation.

However, after reviewing my development and deployment practices, I believe the most likely cause is that a service account credential or API key associated with the project may have been unintentionally exposed in the past. This project has been used for a mobile application backend and Firebase Cloud Messaging (FCM) related development and testing.

I have not intentionally created any unauthorized resources, cryptocurrency mining workloads, automated abuse systems, or any activity that would violate Google Cloud policies. If unauthorized activity occurred, I believe it was performed by a third party using compromised credentials.

Because access to the project is currently restricted, I am unable to verify which credentials or resources may have been affected. If access is restored, I will immediately:

• Audit all service accounts and revoke any existing keys.
• Rotate all credentials and API keys associated with the project.
• Review IAM permissions and remove any unnecessary access.
• Inspect all resources for unauthorized deployments and remove them.
• Implement additional controls to prevent future credential exposure.

If possible, I would appreciate any additional information regarding the specific resource, service, credential, or activity that triggered the violation, as this would help me investigate and remediate the issue more effectively.

Thank you for your time and assistance. I am committed to resolving the issue and ensuring compliance with Google Cloud policies.

Best regards,

TaeHwi Lee

대충 요약 =

형들이 조지라 그래서 조지러 갔는데 막혀있다고 현기증난다구요!

그래서 대충 위와 같이 보내고 사건은 일단락 되었었는데, 약 1주일이 걸렸었더랬다

• 그 이후로 IAM 들어가서 권한 조절하고
• Logging 들어가서 이상한거 있나 살펴보고
• Key Rotation 시키고
• 신기한건 여기서 Gemini API 가 실패율이 95%가 떳었다는 것이다.

그렇게 무심코 넘긴 일이 화근이 될 줄 몰랐다.

대략 2개의 API를 발급해서 사용하고 있었던 것 같은데, 이 때문인지 몰라도 또다시 Violation 으로 인하여 중지되었다는 알림이 5월 말 떠버리기 시작한 것

이번에는 유저들이 단단히 뿔이 낫다는 거시다.

여차저차 이메일을 또쓰고 또쓰고 또쓰고 하는데 해결이 되지 않았고,

결국 나는 마지막 술수를 쓰기에 이르렀는데

이른바 Google Cloud Community에 가서 어그로 끌기 작전이다.

왜 그런고 하니? 내가 왜때문에 처리안해줌? 하면서 티켓을 여러개 따니까 

"님 ㅅㄱ 안해줌. 말이 너무 많음"

이렇게 되면서 스팸을 걸어버려서 내가 이후에 보낸 메일들을 받지 못할 수도 있다는 킹능적 결론이 나왔다.

이후에 기적적으로 봉인 해제에 들어갔고, 금일 부로 FCM Push Notification을 사용할 수 있게되었다.

그리하여 API Key Service 관리를 들어가보니? 왜때문에 예전에 한번 발급하고 사용하지 않는 키가 돌아다니고 있는것이 아닌가?

- 때문에 그걸 지워버렸다

그리고 Logging 을 돌아다니면서 처리하고 Rotate를 시작한다

{
  "insertId": "4381327247975676422",
  "jsonPayload": {
    "remediationLink": "https://cloud.google.com/docs/security/respond-to-abuse-misuse",
    "@type": "type.googleapis.com/google.cloud.abuseevent.logging.v1.AbuseEvent",
    "reinstatementEvent": {},
    "reason": "No new abuse findings on the projects have been detected on the monitored resource, but its state may have been updated (eg: reinstated).",
    "detectionType": "NO_ABUSE",
    "action": "REINSTATE"
  },
  "resource": {
    "type": "abuseevent.googleapis.com/Location",
    "labels": {
      "resource_container": "projects/399395864699",
      "location": "global"
    }
  },
  "timestamp": "2026-06-08T06:23:45.649531945Z",
  "severity": "ALERT",
  "logName": "projects/moyamo-plus/logs/abuseevent.googleapis.com%2Fabuse_events",
  "receiveTimestamp": "2026-06-08T06:23:46.578221266Z"
}

대충 Cloud Logging 에서 찾아보면 위의 경보에서 찾을 수 있는 것인데, 예전에는 무슨일이 있었는지 찾아보았는데

왜때문인지 자기들이 다 처리해서 아모것도 찾을 수 없다는 것이 빅빡침이라는 것.

여하튼 이것으로 더이상 나의 FCM Push notification을 건들지 말아다오..

추가적으로 Gemini API 만을 사용하기 위한 별도의 Google Cloud Console의 Credential 을 만들어주고 바인딩 해줬다. 그 이후에 서버 안에다가 살포시 파킹시킨 뒤에 데몬을 재시작 시켰다

혹시모르니까  FCM의 Topic이 한두개로 고정되있는걸 수정해야 할 지도 모르겠다.

퇴근해서 근근히 스트레스 풀이용으로 돌아가는 나름 자동성장 게임인 그놈은 드래곤..

드디어 용가리가 성장기로 진화했다는 것이다.

다른 유저들은? 분명히 돈을 여러모로 썻겠지만 나는 이걸 모으기 위해 한땀한땀 모아줫다.

저기 중간에 보이는 '비전 중급 진화석' 이 6개가 들어가는데, 모으기 조금 쉬운 팁은

활동은 나름하는데, 길드경매에서 길드경매를 잘 안쓰는 길드를 고르면 된다

그러면 길드경매에서 내가 사고싶은 물품을 독차지 할 수 있다.

신화스킨 조각 3개는 2700 다이아에 이것저것 올라오긴 하는데 그런것들을 죄다 먼저 선점할 수 있다는 것.

다만 약간의 함정 중 하나는, 그런 길드에 들어가면 다른 길드와는 다르게 필드몬스터 사냥도 좀 느릴 것이고

이것저것 아쉬운 점이 많이 생기게 된다.

(그래서 최근에 다른 길드로 넘어왔다지.)

진화를 누르면 왼쪽의 귀여운 킹가리가 아래의 킹가리로 변신한다.

중급 엘프드래곤 조각이 48개정도 있었어서 1성은 바로 뚫어줫는데, 이녀석 살펴보면

작은조각과 중급조각이 이제 동시에 들어간다;; 벌써부터 대 험난한 시대가 예상된다

그런데? 인덱스를 밀어넣고 난 이후에도 데이터가 제대로 보이지 않는다는 것이었다.

흠. 이 부분에 대해서 킹치만 많은 가설을 통해 검증을 시작하면

========================================
ubuntu@ip-172-31-40-155:~$ curl -X GET "http://localhost:9200/_cat/indices?v"
health status index                                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   moyamo_3                                     9bPawqGQTseRptxkTTsy-A   5   1    1576206          774      1.3gb          1.3gb
yellow open   net.infobank.moyamo.models.photorecentwriter bYxm8ozGSzOmC8LpV59oWg   5   1          0            0       810b           810b

내가 예상한 데이터와 다르게 얼마 안들어갔다는 것이다?

이 부분을 계산해볼때 뭔가 이상한 느낌이 스물스물 나서 코드를 조사해봤는데

// SimdaSchedulerApplication.java:122
List<SimplePosting> postings = eventWorkRepositoryCustom.findPostingList(
    Posting.class, sinceId, maxId, 1000  // ← 최대 1000개만 조회!
);

1000개씩만 드신다는 것이었다.

그래서 이걸 진짜로 9~10시간에 걸려서 삽질을 해야되냐? 댓츠논노

우리는 무적의 m5.xlarge를 사용하고 있기 때문에 더욱 공격적으로 병렬로 때려넣을 수 있다는 것이다.

#!/bin/bash
# Elasticsearch 병렬 인덱싱 스크립트
# 8개의 프로세스로 병렬 처리 - m5.xlarge (4 vCPU, 16GB) 최적화
# 예상 소요 시간: 약 1시간

SCHEDULER_URL="http://localhost:8088"
MAX_COMMENTS=200

# ID 범위를 n등분해서 처리
reindex_range() {
    local START=$1
    local END=$2
    local WORKER=$3
    local BATCH_SIZE=1000
    local CURRENT=$START
    local COUNT=0

    echo "[Worker $WORKER] 시작: $START ~ $END"

    while [ $CURRENT -le $END ]; do
        local BATCH_END=$((CURRENT + BATCH_SIZE - 1))
        if [ $BATCH_END -gt $END ]; then
            BATCH_END=$END
        fi

        curl -s -X POST "$SCHEDULER_URL/indexing?sinceId=$CURRENT&maxId=$BATCH_END&max=$MAX_COMMENTS" --max-time 300 > /dev/null

        CURRENT=$((BATCH_END + 1))
        COUNT=$((COUNT + 1))

        # 500 배치마다 진행상황 출력
        if [ $((COUNT % 500)) -eq 0 ]; then
            echo "[Worker $WORKER] 진행 중... $CURRENT / $END"
        fi
    done

    echo "[Worker $WORKER] 완료!"
}

echo "========================================"
echo "병렬 인덱싱 시작 (8 workers)"
echo "서버: m5.xlarge (4 vCPU, 16GB RAM)"
echo "========================================"

START_TIME=$(date +%s)

# 8개의 범위로 나눠서 병렬 실행 (각 약 2백만 건)
reindex_range 114 2000000 1 &
reindex_range 2000001 4000000 2 &
reindex_range 4000001 6000000 3 &
reindex_range 6000001 8000000 4 &
reindex_range 8000001 10000000 5 &
reindex_range 10000001 12000000 6 &
reindex_range 12000001 14000000 7 &
reindex_range 14000001 16335566 8 &

# 모든 백그라운드 작업 완료 대기
wait

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))
HOURS=$((DURATION / 3600))
MINUTES=$(( (DURATION % 3600) / 60 ))
SECS=$((DURATION % 60))

echo "========================================"
echo "전체 인덱싱 완료!"
echo "소요 시간: ${HOURS}시간 ${MINUTES}분 ${SECS}초"
echo "========================================"

# 최종 문서 수 확인
echo "인덱싱된 문서 수:"
curl -s "http://localhost:9200/moyamo_3/_count"
echo """

하지만 이렇게 처리한 이후에 CPU는 영 좋지못한 상황으로 가버리는데

ubuntu@ip-172-31-40-155:~$ chmod +x reindex-parallel.sh 
ubuntu@ip-172-31-40-155:~$ ./reindex-parallel.sh 
========================================
병렬 인덱싱 시작 (8 workers)
서버: m5.xlarge (4 vCPU, 16GB RAM)
========================================
[Worker 1] 시작: 114 ~ 2000000
[Worker 2] 시작: 2000001 ~ 4000000
[Worker 7] 시작: 12000001 ~ 14000000
[Worker 6] 시작: 10000001 ~ 12000000
[Worker 4] 시작: 6000001 ~ 8000000
[Worker 3] 시작: 4000001 ~ 6000000
[Worker 5] 시작: 8000001 ~ 10000000
[Worker 8] 시작: 14000001 ~ 16335566

괘..괜찮아.. 새벽에 돌리는 Scheduler니까 서비스는 영향받지 않는다굿!

문제 상황

대략적으로 서술해보자면 서비스는 일단 502 Bad Gateway가 나면서 내려가고 식은땀이 줄줄 흐르는 상황이 발생한 것인데, 에러 로그만 수집해보면 다음과 같다.

Information Gathering

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
[Rest][2026-03-30 03:49:03][ERROR][SpringApplication.java][reportFailure(860)] : Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:162)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:771)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:763)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:438)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:339)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318)
        at net.infobank.moyamo.MoyamoApplication.main(MoyamoApplication.java:16)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:142)
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104)
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:450)
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:199)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:159)
        ... 17 common frames omitted

그리고 2번째 놈의 로그

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'tokenAuthorizationFilter' defined in URL [jar:file:/home/ubuntu/moyamo-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/net/infobank/moyamo/configuration/TokenAuthorizationFilter.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/home/ubuntu/moyamo-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/net/infobank/moyamo/api/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSecurityRepository' defined in net.infobank.moyamo.repository.UserSecurityRepository defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration: Cannot create inner bean '(inner bean)#59221b97' of type [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean property 'entityManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#59221b97': Cannot resolve reference to bean 'entityManagerFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1354)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:212)
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:175)
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:170)
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAdaptableBeans(ServletContextInitializerBeans.java:155)
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:87)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:259)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:233)
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:53)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5161)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:829)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:433)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486)
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123)
        ... 22 common frames omitted

3번째 로그와 4번째 로그

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/home/ubuntu/moyamo-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/net/infobank/moyamo/api/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSecurityRepository' defined in net.infobank.moyamo.repository.UserSecurityRepository defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration: Cannot create inner bean '(inner bean)#59221b97' of type [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean property 'entityManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#59221b97': Cannot resolve reference to bean 'entityManagerFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1354)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
        ... 63 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSecurityRepository' defined in net.infobank.moyamo.repository.UserSecurityRepository defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration: Cannot create inner bean '(inner bean)#59221b97' of type [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean property 'entityManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#59221b97': Cannot resolve reference to bean 'entityManagerFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:389)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:134)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1689)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1434)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
        ... 77 common frames omitted

로그로그

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#59221b97': Cannot resolve reference to bean 'entityManagerFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:342)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:113)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:693)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:510)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:374)
        ... 91 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:330)
        ... 99 common frames omitted
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:421)
        at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:396)
        at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1845)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782)
        ... 106 common frames omitted

Caused by: org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'moyamo_3'
        at org.hibernate.search.elasticsearch.schema.impl.DefaultElasticsearchSchemaMigrator.migrate(DefaultElasticsearchSchemaMigrator.java:77)
        at org.hibernate.search.elasticsearch.impl.ElasticsearchIndexManager.initializeIndex(ElasticsearchIndexManager.java:321)
        at org.hibernate.search.elasticsearch.impl.ElasticsearchIndexManager.initializeIndex(ElasticsearchIndexManager.java:266)
        at org.hibernate.search.elasticsearch.impl.ElasticsearchIndexManager.setSearchFactory(ElasticsearchIndexManager.java:257)
        at org.hibernate.search.indexes.impl.IndexManagerHolder.setActiveSearchIntegrator(IndexManagerHolder.java:150)
        at org.hibernate.search.engine.impl.MutableSearchFactoryState.setActiveSearchIntegrator(MutableSearchFactoryState.java:244)
        at org.hibernate.search.spi.SearchIntegratorBuilder.buildNewSearchFactory(SearchIntegratorBuilder.java:214)
        at org.hibernate.search.spi.SearchIntegratorBuilder.buildSearchIntegrator(SearchIntegratorBuilder.java:128)
        at org.hibernate.search.hcore.impl.HibernateSearchSessionFactoryObserver.boot(HibernateSearchSessionFactoryObserver.java:127)
        at org.hibernate.search.hcore.impl.HibernateSearchSessionFactoryObserver.sessionFactoryCreated(HibernateSearchSessionFactoryObserver.java:94)
        at org.hibernate.internal.SessionFactoryObserverChain.sessionFactoryCreated(SessionFactoryObserverChain.java:35)
        at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:385)
        at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:468)
        at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1259)
        at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:58)
        at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365)
        at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:409)
        ... 110 common frames omitted
        Suppressed: org.hibernate.search.exception.SearchException: HSEARCH400035: Could not update mappings in index 'net.infobank.moyamo.models.photorecentwriter'
                ... 127 common frames omitted
        Caused by: org.hibernate.search.exception.SearchException: HSEARCH400020: Could not create mapping for entity type net.infobank.moyamo.models.PhotoRecentWriter
                at org.hibernate.search.elasticsearch.schema.impl.ElasticsearchSchemaAccessor.putMapping(ElasticsearchSchemaAccessor.java:111)
                at org.hibernate.search.elasticsearch.schema.impl.DefaultElasticsearchSchemaMigrator.migrate(DefaultElasticsearchSchemaMigrator.java:73)
                ... 126 common frames omitted
        Caused by: org.hibernate.search.exception.SearchException: HSEARCH400007: Elasticsearch request failed.
Request: PUT /net.infobank.moyamo.models.photorecentwriter/net.infobank.moyamo.models.PhotoRecentWriter/_mapping with parameters {}
Response: 400 'Bad Request' with body
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Mapper for [id] conflicts with existing mapping in other types:\n[mapper [id] has different [store] values]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Mapper for [id] conflicts with existing mapping in other types:\n[mapper [id] has different [store] values]"
  },
  "status": 400
}
                at org.hibernate.search.elasticsearch.work.impl.SimpleElasticsearchWork.handleResult(SimpleElasticsearchWork.java:101)
                at org.hibernate.search.elasticsearch.work.impl.SimpleElasticsearchWork.lambda$execute$3(SimpleElasticsearchWork.java:63)
                at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
                at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
                at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
                at org.hibernate.search.elasticsearch.client.impl.DefaultElasticsearchClient$1.onFailure(DefaultElasticsearchClient.java:120)
                at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:857)
                at org.elasticsearch.client.RestClient$1.completed(RestClient.java:560)
                at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
                at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
                at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
                at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
                at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
                at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
                at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
                at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
                at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
                at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
                at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
                at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
                at java.lang.Thread.run(Thread.java:750)
        Caused by: org.hibernate.search.exception.SearchException: HSEARCH400090: Elasticsearch response indicates a failure.
                at org.hibernate.search.elasticsearch.work.impl.DefaultElasticsearchRequestSuccessAssessor.checkSuccess(DefaultElasticsearchRequestSuccessAssessor.java:107)
                at org.hibernate.search.elasticsearch.work.impl.DefaultElasticsearchRequestSuccessAssessor.checkSuccess(DefaultElasticsearchRequestSuccessAssessor.java:90)
                at org.hibernate.search.elasticsearch.work.impl.SimpleElasticsearchWork.handleResult(SimpleElasticsearchWork.java:92)
                ... 24 common frames omitted
Caused by: org.hibernate.search.exception.SearchException: HSEARCH400020: Could not create mapping for entity type net.infobank.moyamo.models.AdoptComment
        at org.hibernate.search.elasticsearch.schema.impl.ElasticsearchSchemaAccessor.putMapping(ElasticsearchSchemaAccessor.java:111)
        at org.hibernate.search.elasticsearch.schema.impl.DefaultElasticsearchSchemaMigrator.migrate(DefaultElasticsearchSchemaMigrator.java:73)
        ... 126 common frames omitted
Caused by: org.hibernate.search.exception.SearchException: HSEARCH400007: Elasticsearch request failed.
Request: PUT /moyamo_3/net.infobank.moyamo.models.AdoptComment/_mapping with parameters {}
Response: 400 'Bad Request' with body
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "mapper [relation.posting.title] of different type, current_type [text], merged_type [keyword]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "mapper [relation.posting.title] of different type, current_type [text], merged_type [keyword]"
  },
  "status": 400
}


        at org.hibernate.search.elasticsearch.work.impl.SimpleElasticsearchWork.handleResult(SimpleElasticsearchWork.java:101)
        at org.hibernate.search.elasticsearch.work.impl.SimpleElasticsearchWork.lambda$execute$3(SimpleElasticsearchWork.java:63)
        at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
        at org.hibernate.search.elasticsearch.client.impl.DefaultElasticsearchClient$1.onFailure(DefaultElasticsearchClient.java:120)
        at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:857)
        at org.elasticsearch.client.RestClient$1.completed(RestClient.java:560)
        at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.hibernate.search.exception.SearchException: HSEARCH400090: Elasticsearch response indicates a failure.
        at org.hibernate.search.elasticsearch.work.impl.DefaultElasticsearchRequestSuccessAssessor.checkSuccess(DefaultElasticsearchRequestSuccessAssessor.java:107)
        at org.hibernate.search.elasticsearch.work.impl.DefaultElasticsearchRequestSuccessAssessor.checkSuccess(DefaultElasticsearchRequestSuccessAssessor.java:90)
        at org.hibernate.search.elasticsearch.work.impl.SimpleElasticsearchWork.handleResult(SimpleElasticsearchWork.java:92)
        ... 24 common frames omitted

대충 상황요약

HSEARCH400035: Could not update mappings in index 'moyamo_3'
Mapper for [id] conflicts with existing mapping in other types:
[mapper [id] has different [store] values]

ElasticSearch + Hibernate Search 관련 매핑 충돌 에러였다.

이게 왜 문제가 발생했냐면 PhotoRecentWriter 엔티티 id 필드 매핑이 기존 ElasticSearch 인덱스 매핑이랑 충돌한 것인데.

쥔짜 식은땀을 흘렷는데 예전 개발자녀석 왜 이렇게 짜놧는지 이해가 안감. 대충 아래의 소스를 살펴보면

package net.infobank.moyamo.models;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.NonNull;
import org.hibernate.search.annotations.Indexed;
import org.hibernate.search.annotations.NumericField;

import javax.persistence.*;
import java.time.ZonedDateTime;

@Data
@Entity
@AllArgsConstructor
@NoArgsConstructor
@Indexed
public class PhotoRecentWriter {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @NumericField
    private Long id;

    @NonNull
    @Embedded
    private UserPostingRelation relation;

    private ZonedDateTime createdAt;

    public PhotoRecentWriter(@NonNull UserPostingRelation userPostingRelation, ZonedDateTime createdAt){
        this.relation = userPostingRelation;
        this.createdAt = createdAt;
    }
}

이 부분에서 저기 보이는 Indexed 어노테이션을 사용하면서 생기는 문제였다. 인덱스 이름을 지정하지 않아서 생기는 문제인 것인데, 그래서 기본 클래스명인 net.infobank.moyamo.models.photorecentwriter 으로 별도 인덱스가 만들어졌던거임.

근데? 에러 로그를 보면 다음과 같다.

Could not update mappings in index 'moyamo_3'

moyamo_3 인덱스 내에서 PhotoRecentWriter의 id 필드가 다른 엔티티 의 id 필드 매핑이랑 store 값이 충돌하기 때문

대충 차이점을 잡고가자면 이런 부분이다

• User.java, Posting.java 의 id:@NumericFiled + @SortableField (store 속성 없음 = 기본값 Store.NO)
• PhotoRecentWriter.java의 id:@NumericField만 있다는 것

이 부분을 해결하려는 방법은 되게 간단했다.

PhotoRecentWriter의 id 필드에 동일한 어노테이션을 적용하거나, 아예 ElasticSearch 인덱싱이 필요 없으면 Indexed 어노테이션을 제거하면 된다.

@Data
@Entity
@AllArgsConstructor
@NoArgsConstructor
public class PhotoRecentWriter {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @NonNull
    @Embedded
    private UserPostingRelation relation;

    private ZonedDateTime createdAt;

    public PhotoRecentWriter(@NonNull UserPostingRelation userPostingRelation, ZonedDateTime createdAt){
        this.relation = userPostingRelation;
        this.createdAt = createdAt;
    }
}

실제로 PhotoRecentWriter 가 검색기능에 사용되지 않으므로 제거하면되는 부분이므로 JPA엔티니니까 DB에서만 쓰는거임 ㅇㅇ

그런데? 난 사실 이걸 먼저하기보다 이 짓을 먼저했음

curl -X DELETE "localhost:9200/net.infobank.moyamo.models.photorecentwriter"

그러니까 502 Bad Gateway는 해결되서 올라오긴 하는데, 사람들이 예전에 쓴 글이 안올라온다고 하는 대략난감의 사태가 벌어졌다는 거쉼..

이게 영향이 있냐 없냐 고민을 하다보니 100% 있다는 것임!

그렇게 시작되었다. Hibernate Search에서 기존 DB 데이터를 ElasticSearch에 다시 인덱싱하는 개삽질의 작업

식은땀을 줄줄 흘리다가. 나자신 생각했다. 하늘이 무너져도 솟아날 구멍? 댓츠논노

어쨋든 저놈들은 죄다 ElasticSearch 기반의 작업일꺼니까 인덱스만 다시 치면 된다는 생각이었음.

솔직히 JPA 안에 본체가 있는데 ElasticSearch가 캐싱하는 데이터 구도라면? 캐싱만 치면 되는 부분임

심지어 찾아보니 땅굴을 좀 파놧는데 이런 것들이 있었음

1. Posting 인덱스 재구축

# POST 요청 - sinceId부터 maxId까지 게시글 인덱싱
curl -X POST "http://<scheduler-server>/indexing?sinceId=0&maxId=999999999&max=200"

또는 api-server 에서

# 특정 게시글 ID들 인덱싱
curl -X POST "http://<api-server>/v2/postings/index?ids=1,2,3,4,5&max=200"

2. User 인덱스 재구축

# moyamo-api-server에서
curl -X GET "http://<api-server>/v2/users/index"

그런데 아무리 생각해봐도 API에다가 저짓을 하다가는 서버가 죽을 것이 뻔했음.

MySQL 에서 데이터를 조사함

-- MySQL에서 게시글 ID 범위 확인
SELECT MIN(id), MAX(id) FROM posting WHERE is_delete = false;

참으로 기모찌한 결과가 나온 것인데, 이걸 curl 로 나눠서 쏘다가는 꽤 귀찮을 것 같다는 거쉬..

그래서 인덱스 스크립트를 만들어서 때려박기로 했다.

#!/bin/bash
# Elasticsearch 인덱스 재구축 스크립트 (Linux/Mac용)
# 게시글 ID 범위: 114 ~ 16,335,566

SCHEDULER_URL="http://localhost:8088"  # moyamo-scheduler 포트: 8088
BATCH_SIZE=10000                        # 한 번에 처리할 ID 범위
MAX_COMMENTS=200                        # 댓글 수 제한
DELAY_SECONDS=2                         # 배치 간 대기 시간

MIN_ID=114
MAX_ID=16335566

CURRENT_ID=$MIN_ID
TOTAL_BATCHES=$(( (MAX_ID - MIN_ID + BATCH_SIZE - 1) / BATCH_SIZE ))
CURRENT_BATCH=0

echo "========================================"
echo "Elasticsearch 인덱스 재구축 시작"
echo "총 배치 수: $TOTAL_BATCHES"
echo "ID 범위: $MIN_ID ~ $MAX_ID"
echo "배치 크기: $BATCH_SIZE"
echo "========================================"

START_TIME=$(date +%s)

while [ $CURRENT_ID -le $MAX_ID ]; do
    CURRENT_BATCH=$((CURRENT_BATCH + 1))
    END_ID=$((CURRENT_ID + BATCH_SIZE - 1))
    if [ $END_ID -gt $MAX_ID ]; then
        END_ID=$MAX_ID
    fi

    PROGRESS=$(echo "scale=1; $CURRENT_BATCH * 100 / $TOTAL_BATCHES" | bc)
    echo "[$CURRENT_BATCH/$TOTAL_BATCHES] ($PROGRESS%) 인덱싱 중: $CURRENT_ID ~ $END_ID"

    RESPONSE=$(curl -s -X POST "$SCHEDULER_URL/indexing?sinceId=$CURRENT_ID&maxId=$END_ID&max=$MAX_COMMENTS" --max-time 300)

    if [ $? -eq 0 ]; then
        echo "  완료: $RESPONSE"
    else
        echo "  오류 발생, 재시도 중..."
        sleep 5
        RESPONSE=$(curl -s -X POST "$SCHEDULER_URL/indexing?sinceId=$CURRENT_ID&maxId=$END_ID&max=$MAX_COMMENTS" --max-time 300)
        if [ $? -eq 0 ]; then
            echo "  재시도 완료: $RESPONSE"
        else
            echo "  재시도 실패, 다음 배치로 진행"
        fi
    fi

    CURRENT_ID=$((END_ID + 1))

    if [ $CURRENT_ID -le $MAX_ID ]; then
        sleep $DELAY_SECONDS
    fi
done

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))
HOURS=$((DURATION / 3600))
MINUTES=$(( (DURATION % 3600) / 60 ))
SECONDS=$((DURATION % 60))

echo "========================================"
echo "인덱싱 완료!"
echo "소요 시간: ${HOURS}시간 ${MINUTES}분 ${SECONDS}초"
echo "========================================"

일단 테스트도 완료했겠다 신나게 돌려본다는 것인데..

그 전에 주머니에서 주섬주섬 moyamo-common 부터 말은 이후에 api-server를 말아서 build.jar로 만든다음에 배포 했다는 스토리는 생략하도록 하고, 저게 돌아가는건 결국 스케쥴러가 돌아가는 인스턴스에서 돌아가고 있는 부분이었으니까 포트부터 조사하면

스크립트 → moyamo-scheduler → Elasticsearch
              (/indexing API)      (내부적으로 연결)

이런 구조가 나온다는 것이었다.

생각해보니 Tools 서버를 껏다키면서 ElasticSearch 때문에 잘 안올라오는 다는 부분을 생각해보면서 볼륨 detach 했다가 다시 attach 하고 그런 부분 와중에 데몬을 제대로 안살렸었더라는 기억이 있었으므로 스케쥴러부터 살려줌

ubuntu@ip-172-31-40-155:~/workspace/scheduler/logs$ ps -ef | grep scheduler
ubuntu   15665  6442  0 12:34 pts/1    00:00:00 grep --color=auto scheduler
ubuntu@ip-172-31-40-155:~/workspace/scheduler/logs$ cd ..
ubuntu@ip-172-31-40-155:~/workspace/scheduler$ ls -alk
total 40
drwxrwxr-x 5 ubuntu ubuntu  4096 Oct  7  2024 .
drwxrwxr-x 4 ubuntu ubuntu  4096 Mar 30  2021 ..
drwxrwxr-x 2 ubuntu ubuntu  4096 Oct 21  2024 bin
drwxrwxr-x 2 ubuntu ubuntu 16384 Mar 30 12:32 logs
-rwxrwxr-x 1 ubuntu ubuntu   253 Oct 13  2023 start.sh
drwxrwxr-x 3 ubuntu ubuntu  4096 Sep  7  2020 static
-rwxrwxr-x 1 ubuntu ubuntu    58 Apr 16  2021 stop.sh
ubuntu@ip-172-31-40-155:~/workspace/scheduler$ vi start.sh 
ubuntu@ip-172-31-40-155:~/workspace/scheduler$ vi start.sh 
ubuntu@ip-172-31-40-155:~/workspace/scheduler$ ./start.sh 
ubuntu@ip-172-31-40-155:~/workspace/scheduler$ ps -ef | grep scheduler
ubuntu   18233     1 99 12:36 pts/1    00:00:11 java -Xmx1500m -Xms1500m -jar -Dspring.profiles.active=product -Dmoyamo.jobs.rankings.cron=0 */1 * * * * -Dmoyamo.jobs.statistics.cron=0 */1 * * * * /home/ubuntu/workspace/scheduler/bin/moyamo-scheduler-0.0.1-SNAPSHOT.jar
ubuntu   18373  6442  0 12:36 pts/1    00:00:00 grep --color=auto scheduler

이제 언제 살아날지 모르는 대망의 작업을 시작한다.

ubuntu@ip-172-31-40-155:~$ chmod +x reindex-elasticsearch.sh 
ubuntu@ip-172-31-40-155:~$ ./reindex-elasticsearch.sh 
========================================
Elasticsearch 인덱스 재구축 시작
총 배치 수: 1634
ID 범위: 114 ~ 16335566
배치 크기: 10000
========================================
[1/1634] (0%) 인덱싱 중: 114 ~ 10113
  완료: end
[2/1634] (.1%) 인덱싱 중: 10114 ~ 20113
  완료: end
[3/1634] (.1%) 인덱싱 중: 20114 ~ 30113
  완료: end
[4/1634] (.2%) 인덱싱 중: 30114 ~ 40113
  완료: end
[5/1634] (.3%) 인덱싱 중: 40114 ~ 50113
  완료: end
[6/1634] (.3%) 인덱싱 중: 50114 ~ 60113
  완료: end
[7/1634] (.4%) 인덱싱 중: 60114 ~ 70113
  완료: end
[8/1634] (.4%) 인덱싱 중: 70114 ~ 80113
  완료: end
[9/1634] (.5%) 인덱싱 중: 80114 ~ 90113
  완료: end
[10/1634] (.6%) 인덱싱 중: 90114 ~ 100113
  완료: end
[11/1634] (.6%) 인덱싱 중: 100114 ~ 110113
  완료: end
[12/1634] (.7%) 인덱싱 중: 110114 ~ 120113
  완료: end
[13/1634] (.7%) 인덱싱 중: 120114 ~ 130113
  완료: end
[14/1634] (.8%) 인덱싱 중: 130114 ~ 140113
  완료: end
[15/1634] (.9%) 인덱싱 중: 140114 ~ 150113
  완료: end
[16/1634] (.9%) 인덱싱 중: 150114 ~ 160113