/images/profile.png

์†Œ๋‚˜ํ๋ธŒ ์ด์šฉ ์ฝ”๋“œ ์ •์ ๋ถ„์„ ์ž๋™ํ™”

์ฝ”๋“œ ์ •์ ๋ถ„์„์ด๋ผ ํ•จ์€ ์‹ค์ œ ํ”„๋กœ๊ทธ๋žจ์„ ์‹คํ–‰ํ•˜์ง€ ์•Š๊ณ  ์ฝ”๋“œ๋งŒ์˜ ํ˜•ํƒœ์— ๋Œ€ํ•œ ๋ถ„์„์„ ๋งํ•œ๋‹ค. ์ด๋ฅผํ…Œ๋ฉด ๋ƒ„์ƒˆ๋‚˜๋Š” ์ฝ”๋“œ(?)๋ผ๋˜์ง€, ์œ„ํ—˜์„ฑ์ด ์žˆ๋Š” ์ฝ”๋“œ, ๋ฏธ๋ฆฌ ์ •์˜๋œ ๊ทœ์น™์ด๋‚˜ ์ฝ”๋”ฉ ํ‘œ์ค€์„ ์ค€์ˆ˜ํ•˜๋Š”์ง€์— ๋Œ€ํ•œ ๋ถ„์„์„ ๋งํ•˜๋Š”๋ฐ java ๊ธฐ์ค€์œผ๋กœ๋Š” ์•„๋ž˜ ๋‹ค์–‘ํ•œ (์ž˜ ์•Œ๋ ค์ง„) ์ •์ ๋ถ„์„ ๋„๊ตฌ๋“ค์ด ์žˆ๋‹ค.

  • PMD
    • ๋ฏธ์‚ฌ์šฉ ๋ณ€์ˆ˜, ๋น„์–ด์žˆ๋Š” ์ฝ”๋“œ ๋ธ”๋ฝ, ๋ถˆํ•„์š”ํ•œ ์˜ค๋ธŒ์ ํŠธ ์ƒ์„ฑ๊ณผ ๊ฐ™์€ Defect์„ ์œ ๋ฐœํ•  ์ˆ˜ ์žˆ๋Š” ์ฝ”๋“œ๋ฅผ ๊ฒ€์‚ฌ
    • https://pmd.github.io
  • FindBugs
    • ์ •ํ•ด์ง„ ๊ทœ์น™์— ์˜ํ•ด ์ž ์žฌ์ ์ธ ์—๋Ÿฌ ํƒ€์ž…์„ ์ฐพ์•„์คŒ
    • http://findbugs.sourceforge.net
  • CheckStyle
    • ์ •ํ•ด์ง„ ์ฝ”๋”ฉ ๋ฃฐ์„ ์ž˜ ๋”ฐ๋ฅด๊ณ  ์žˆ๋Š”์ง€์— ๋Œ€ํ•œ ๋ถ„์„
    • http://checkstyle.sourceforge.net

์ด์™ธ์— SonarQube ๋ผ๋Š” ํˆด์ด ์žˆ๋Š”๋ฐ ๊ฐœ์ธ์ ์œผ๋กœ ์œ„ ์•Œ๋ ค์ง„ ๋‹ค๋ฅธ ํˆด๋“ค์˜ ์ข…ํ•ฉํŒ(?)์ด๋ผ๊ณ  ์ƒ๊ฐ์ด ๋“ค์—ˆ๊ณ , ๊ทธ์ค‘ ๊ฐ€์žฅ ์ธ์ƒ๊นŠ์—ˆ๋˜ ๊ธฐ๋Šฅ์ด github๊ณผ ์—ฐ๋™์ด ๋˜๊ณ  ์ ์ ˆํ•œ ๊ตฌ์„ฑ์„ ํ•˜๊ฒŒ ๋˜๋ฉด ์ฝ”๋“œ๋ฅผ ์ˆ˜์ •ํ•˜๋Š”๊ณผ ๋™์‹œ์— ์ž๋™์œผ๋กœ ๋ถ„์„์„ ํ•˜๊ณ  ๋ฆฌํฌํŒ…๊นŒ์ง€ ํ•ด์ค€๋‹ค๋Š” ๋ถ€๋ถ„์ด์˜€๋‹ค. ( ๋” ์ข‹์€ ๋ฐฉ๋ฒ•์ด ์žˆ๋Š”์ง€๋Š” ๋ชจ๋ฅด๊ฒ ์œผ๋‚˜ ๋‹ค๋ฅธ ๋„๊ตฌ๋“ค์€ ์ˆ˜๋™์œผ๋กœ ๋Œ๋ ค์ค˜์•ผ ํ•˜๊ณ  ๋ฆฌํฌํŒ… ๋˜ํ•œ Activeํ•˜์ง€ ๋ชปํ•œ(?) ์•„์‰ฌ์šด ์ ์ด ์žˆ์—ˆ๋‹ค. )

์ง€๊ธˆ๋ถ€ํ„ฐ Jenkins + github web-hook + SonarQube ๋ฅผ ๊ตฌ์„ฑํ•˜์—ฌ ์ฝ”๋“œ๋ฅผ ์ˆ˜์ •ํ•˜๊ณ  PullRequest๋ฅผ ์˜ฌ๋ฆฌ๊ฒŒ ๋˜๋ฉด ์ˆ˜์ •ํ•œ ํŒŒ์ผ์— ๋Œ€ํ•ด ์ž๋™์œผ๋กœ ์ •์ ๋ถ„์„์ด ์ด๋ค„์ง€๊ณ , ๊ทธ์—๋Œ€ํ•œ ๋ฆฌํฌํŒ…์ด ํ•ด๋‹น PullRequest์— ๋Œ“๊ธ€๋กœ ๋‹ฌ๋ฆฌ๋„๋ก ์„ค์ •์„ ํ•ด๋ณด๊ฒ ๋‹ค. (์ฝ”๋“œ๋ฆฌ๋ทฐ๋ฅผ ๋ด‡(?)์ด ์ž๋™์œผ๋กœ ํ•ด์ฃผ๋Š”๊ฒŒ ์–ผ๋งˆ๋‚˜ ํŽธํ•œ ์ผ์ธ๊ฐ€…)

๊ธฐ๋ณธ ์ปจ์…‰

์ „์ฒด์ ์ธ ์ปจ์…‰์€ ๋‹ค์Œ ๊ทธ๋ฆผ๊ณผ ๊ฐ™๋‹ค.

/images/jenkins-sonar-github-integration/concept.png
์ „์ฒด ์ปจ์…‰
  1. IDE์—์„œ ์ฝ”๋“œ์ˆ˜์ •์„ ํ•˜๊ณ  remote ์ €์žฅ์†Œ์— commit & push๋ฅผ ํ•œ๋‹ค. ๊ทธ ๋‹ค์Œ github์—์„œ master(ํ˜น์€ stableํ•œ branch)์— ๋Œ€ํ•ด ์ž‘์—… branch๋ฅผ PullRequest ์˜ฌ๋ฆฐ๋‹ค.
  2. ๋ฏธ๋ฆฌ ๋“ฑ๋กํ•œ github์˜ web-hook์— ์˜ํ•ด PullRequest ์ •๋ณด๋“ค์„ jenkins์— ์ „์†กํ•œ๋‹ค.
  3. ์ „๋‹ฌ๋ฐ›์€ ์ •๋ณด๋ฅผ ์žฌ ๊ฐ€๊ณตํ•˜์—ฌ SonarQube๋กœ ์ •์ ๋ถ„์„์„ ์š”์ฒญํ•œ๋‹ค.
  4. SonarQube์—์„œ ๋ถ„์„ํ•œ ์ •๋ณด๋ฅผ ๋‹ค์‹œ jenkins๋กœ return ํ•ด์ค€๋‹ค.
  5. SonarQube์œผ๋กœ๋ถ€ํ„ฐ return ๋ฐ›์€ ์ •๋ณด๋ฅผ ํ•ด๋‹น PullRequest์˜ ๋Œ“๊ธ€์— ๋ฆฌํฌํŒ…์„ ํ•ด์ค€๋‹ค.

๊ฐ„๋‹จํžˆ ๋ณด๋ฉด (๋ญ ๊ฐ„๋‹จํ•˜๋‹ˆ ์‰ฝ๋„ค~) ๋ผ๊ณ  ๋ณผ์ˆ˜๋„ ์žˆ๊ฒ ์ง€๋งŒ ๋‚˜๋Š” ์ด๋Ÿฐ ์ „์ฒด ํ๋ฆ„์„ ์„ค์ •ํ•˜๋Š”๋ฐ ์žˆ์–ด ์–ด๋ ค์› ๋‹ค.

์‚ฌ์‹ค ์…‹ํŒ…ํ•˜๋Š” ๊ณผ์ •์—์„œ ์ ์ง€์•Š์€ ์‚ฝ์งˆ์„ ํ–ˆ์—ˆ๊ธฐ์—, ์ด ํฌ์ŠคํŒ…์„ ์ ๋Š” ์ด์œ ์ผ์ˆ˜๋„ ์žˆ๊ฒ ๋‹ค. ๋”๋ถˆ์–ด ๊ฒ€์ƒ‰์„ ํ•ด๋ด๋„ ์ด๋ ‡๊ฒŒ ์ „์ฒดํ๋ฆ„์ด ์ •๋ฆฌ๋œ ๊ธ€์ด ์ž˜ ์•ˆ๋ณด์—ฌ์„œ + ๋‚ด๊ฐ€ ํ•œ ์‚ฝ์งˆ์„ ๋‹ค๋ฅธ ๋ˆ„๊ตฐ๊ฐ€๋„ ํ• ๊ฒƒ๊ฐ™์•„์„œ(?)

Maven ์„ค์น˜

๊ธฐ๋ณธ์ ์œผ๋กœ Maven์˜ H2DB๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ SonarQube๋ฅผ ์„ค์น˜ํ•˜๊ธฐ์ „์— Maven๋ถ€ํ„ฐ ์„ค์น˜ํ•ด์ค˜์•ผ ํ•œ๋‹ค.

$ wget http://apache.mirror.cdnetworks.com/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz
$ tar -zxvf apache-maven-3.5.2-bin.tar.gz
(ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์…‹ํŒ…ํ›„ )
$  mvn -version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T16:58:13+09:00)
...

SonarQube ์„ค์น˜

์ •์ ๋ถ„์„์„ ๋„์™€์ฃผ๋Š” SonarQube๋ฅผ ์„ค์น˜ํ•ด๋ณด์ž.

$ wget https://sonarsource.bintray.com/Distribution/sonarqube/sonarqube-6.7.1.zip
$ unzip sonarqube-6.7.1.zip
$ cd sonarqube-6.7.1/bin/linux-x86-64
$ ./sonar.sh start
Starting SonarQube...
Started SonarQube.

๊ธฐ๋ณธ์ ์œผ๋กœ 9000ํฌํŠธ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ์œผ๋‹ˆ ๋‹ค๋ฅธํฌํŠธ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ ์ž ํ•œ๋‹ค๋ฉด /sonarqube-6.7.1/conf/sonar.properties ๋‚ด sonar.web.port=9000 ์„ ์ˆ˜์ •ํ•ด์ฃผ๋ฉด ๋œ๋‹ค. (SonarQube๋„ Elasticsearch๋ฅผ ์‚ฌ์šฉํ•˜๊ตฌ๋‚˜…) ์„ค์น˜ํ›„ ์‹คํ–‰์„ ํ•œ๋’ค ์„œ๋ฒ„IP:9000์„ ์ ‘์†ํ•ด๋ณด๋ฉด ์•„๋ž˜ ํ™”๋ฉด์ฒ˜๋Ÿผ ๋‚˜์˜จ๋‹ค. (ํ˜น์‹œ ์ ‘์†์ด ์•ˆ๋œ๋‹ค๊ฑฐ๋‚˜ ์„œ๋ฒ„๊ฐ€ ์‹คํ–‰์ด ์•ˆ๋œ๋‹ค๋ฉด ./sonar.sh console๋กœ ๋กœ๊ทธ๋ฅผ ๋ณด๋ฉด ๋ฌธ์ œํ•ด๊ฒฐ์— ๋„์›€์ด ๋ ์ˆ˜๋„ ์žˆ๋‹ค. )

/images/jenkins-sonar-github-integration/sonar_main.png
SonarQube ๋ฉ”์ธํ™”๋ฉด

SonarQube Scanner ์„ค์น˜

์†Œ์Šค๋ฅผ ์—ฐ๋™์‹œ์ผœ ์ •์ ๋ถ„์„์„ ํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” SonarQube Scanner ๋ผ๋Š”๊ฒŒ ํ•„์š”ํ•˜๋‹ค๊ณ  ํ•œ๋‹ค. ์•„๋ž˜ url์—์„œ ๋‹ค์šด๋ฐ›์•„ ์ ์ ˆํ•œ ๊ณณ์— ์••์ถ•์„ ํ’€์–ด๋‘์ž. https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner

$ wget https://sonarsource.bintray.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-3.0.3.778-linux.zip
$ unzip sonar-scanner-cli-3.0.3.778-linux.zip

# jenkins ์„ค์น˜ ๋ฐ SonarQube ์—ฐ๋™

jenkins ์„ค์น˜๋Š” ๊ฐ„๋‹จํ•˜๋‹ˆ ๋ณ„๋„ ์–ธ๊ธ‰์€ ์•ˆํ•˜๊ณ  ๋„˜์–ด๊ฐ€…๋ ค๊ณ  ํ–ˆ์œผ๋‚˜, ํ•˜๋‚˜๋ถ€ํ„ฐ ์—ด๊นŒ์ง€ ์ •๋ฆฌํ•œ๋‹ค๋Š” ๋งˆ์Œ์œผ๋กœ~ https://jenkins.io/download/ ์—์„œ ์ตœ์‹ ๋ฒ„์ „์„ tomcat/webapps/ ์•„๋ž˜์— ๋‹ค์šด๋ฐ›๊ณ  server.xml ์„ ์ ์ ˆํ•˜๊ฒŒ ์ˆ˜์ •ํ•ด์ค€๋‹ค.

$ wget http://mirrors.jenkins.io/war-stable/latest/jenkins.war
$ vi tomcat/conf/server.xml
<Connector port="19001" protocol="HTTP/1.1" # ํฌํŠธ ๋ณ€๊ฒฝ
<Context path="/jenkins" debug="0" privileged="true" docBase="jenkins.war" /> #์ถ”๊ฐ€
# tomcat/bin/startup.sh

jenkins ์„ค์น˜๋ฅผ ์™„๋ฃŒ ํ•œ ํ›„ ํ•„์š”ํ•œ ํ”Œ๋Ÿฌ๊ทธ์ธ์„ ์ถ”๊ฐ€๋กœ ์„ค์น˜ํ•ด์ค€๋‹ค.

  • Python Plugin
  • GitHub Pull Request Builder
  • GitHub plugin

์ ‘์† : ์„œ๋ฒ„IP:19001 (์ฐธ๊ณ ๋กœ ํ•œ ์„œ๋ฒ„์—์„œ ๋‹ค ์„ค์น˜ํ•˜๋‹ค๋ณด๋‹ˆ port ์ถฉ๋Œ์„ ์‹ ๊ฒฝ์“ฐ๊ฒŒ๋˜์—ˆ๋‹ค. ) ์ฒ˜์Œ jenkins๋ฅผ ์‹คํ–‰ํ•˜๋ฉด ์ด๋Ÿฐ์ €๋Ÿฐ ์„ค์ •์„ ํ•˜๋Š”๋ฐ ํŠน๋ณ„ํ•œ ์„ค์ • ๋ณ€๊ฒฝ์—†์ด next๋ฒ„ํŠผ์„ ์—ฐ์‹  ๋ˆŒ๋Ÿฌ๋ฉด ์„ค์น˜๊ฐ€ ์™„๋ฃŒ ๋˜๊ณ , SonarQube๋ฅผ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด SonarQube Scanner for Jenkins๋ผ๋Š” ํ”Œ๋Ÿฌ๊ทธ์ธ์„ ์„ค์น˜ํ•ด์ฃผ์ž. (์ด๊ฑด ๊ฐ ๋ฒ„์ „๋งˆ๋‹ค ๊ถํ•ฉ(?)์ด ์•ˆ๋งž์„์ˆ˜๋„ ์žˆ์œผ๋‹ˆ ํ™•์ธ์ด ํ•„์š”ํ• ์ˆ˜๋„ ์žˆ๋‹ค. ๋‚ด๊ฐ€ ์„ค์น˜ํ•œ ๋ฒ„์ „์€ jenkins 2.89, SonarQube Plugin 2.6.1์ด๋‹ค.) ์„ค์น˜๋ฅผ ํ•˜๋ฉด jenkins > configure ์—์„œ SonarQube servers์ •๋ณด๋ฅผ ๋“ฑ๋กํ•ด์ค€๋‹ค.

Github์˜ WebHook์„ ์ด์šฉํ•˜์—ฌ ์ž๋™ Jenkins Job ์‹คํ–‰

PullRequest๊ฐ€ ๋ฐœ์ƒํ•˜๋ฉด ์•Œ๋ฆผ์„ ๋ฐ›๊ณ ์‹ถ๋‹ค๊ฑฐ๋‚˜, ๋‚ด๊ฐ€ ๊ด€๋ฆฌํ•˜๋Š” ๋ ˆํŒŒ์ง€ํ† ๋ฆฌ์— ๋Œ“๊ธ€์ด ๋‹ฌ๋ฆด๋•Œ๋งˆ๋‹ค ๋˜๋Š” ์ด์Šˆ๊ฐ€ ์ƒ์„ฑ๋ ๋•Œ๋งˆ๋‹ค ์ •๋ณด๋ฅผ ์ €์žฅํ•˜๊ณ  ์‹ถ๋‹ค๊ฑฐ๋‚˜. ์ข…ํ•ฉํ•ด๋ณด๋ฉด Github์—์„œ ์ด๋ฒคํŠธ๊ฐ€ ๋ฐœ์ƒํ• ๋•Œ ์–ด๋–ค ๋™์ž‘์„ ํ•ด์•ผ ํ•  ๊ฒฝ์šฐ Github์—์„œ ์ œ๊ณตํ•˜๋Š” Webhook ์„ ์‚ฌ์šฉํ•˜์—ฌ ๋ชฉ์ ์„ ๋‹ฌ์„ฑํ•  ์ˆ˜ ์žˆ๋‹ค. ์•„ ๋‹น์—ฐํ•œ ์ด์•ผ๊ธฐ์ด์ง€๋งŒ ์–ธ๊ธ‰ํ•˜๊ณ  ๋„˜์–ด๊ฐˆ๊ป˜ ์žˆ๋‹ค๋ฉด, Github์—์„œ Jenkins Job์„ ํ˜ธ์ถœํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” Jenkins๊ฐ€ ์™ธ๋ถ€์— ๊ณต๊ฐœ๋˜์–ด ์žˆ์–ด์•ผ ํ•œ๋‹ค. (๋‚ด๋ถ€์‚ฌ์„ค๋ง์ด๋‚˜ private ํ•œ ์„ค์ •์ด ๋˜์–ด์žˆ๋‹ค๋ฉด ํ˜ธ์ถœ์ด ์•ˆ๋˜์–ด Webhook๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†๋‹ค.)

Jenkins Security ์„ค์ •

Jenkins Job์„ ์™ธ๋ถ€์—์„œ URL๋กœ ์‹คํ–‰์„ ํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” ์•„๋ž˜ ์„ค์ •์ด ๊ผญ ํ•„์š”ํ•˜๋‹ค. (์ด ์„ค์ •์„ ๋ชฐ๋ผ์„œ ์ˆ˜๋งŽ์€ ์‚ฝ์งˆ์„ ํ–ˆ๋‹ค.) CSRF Protection ์„ค์ • ์ฒดํฌ๋ฅผ ํ’€์–ด์ค˜์•ผ ํ•œ๋‹ค. ์ด๋ ‡๊ฒŒ ๋˜๋ฉด ์™ธ๋ถ€์—์„œ Job์— ๋Œ€ํ•œ ํŠธ๋ฆฌ๊ฑฐ๋ง์ด ๊ฐ€๋Šฅํ•ด ์ง„๋‹ค.

/images/github-web-hook-jenkins-job-excute/jenkins_config.png
Jenkins > Configure Global Security

Jenkins Job ์„ค์ •

Github ์—์„œ Webhook์— ์˜ํ•ด Jenkins Job์„ ์‹คํ–‰ํ•˜๊ฒŒ ๋ ํ…๋ฐ, ๊ทธ๋•Œ ์ •๋ณด๋“ค์ด payload๋ผ๋Š” ํŒŒ๋ผ๋ฏธํ„ฐ์™€ ํ•จ๊ป˜ POST ํ˜•์‹์œผ๋กœ ํ˜ธ์ถœ์ด ๋˜๊ธฐ ๋•Œ๋ฌธ์— ๋ฏธ๋ฆฌ Job์—์„œ ๋ฐ›๋Š” ์ค€๋น„(?)๋ฅผ ํ•ด๋‘ฌ์•ผ ํ•œ๋‹ค. ์„ค์ •์€ ๊ฐ„๋‹จํ•˜๊ฒŒ ๋‹ค์Œ๊ณผ ๊ฐ™์ด Job ํŒŒ๋ผ๋ฏธํ„ฐ ์„ค์ •์„ ํ•ด์ฃผ๋ฉด ๋œ๋‹ค.

/images/github-web-hook-jenkins-job-excute/jenkins_job_param.png
Jenkins > ํ•ด๋‹น Job > configure

Github Webhook ์„ค์ •

์ด์ œ Github Repository ์˜ Hook ์„ค์ •๋งŒ ํ•˜๋ฉด ๋์ด๋‚œ๋‹ค. ํ•ด๋‹น Repository > Settings > Hooks ์„ค์ •์— ๋“ค์–ด๊ฐ€์„œ Add webhook์„ ์„ ํƒํ•˜์—ฌ Webhook์„ ๋“ฑ๋กํ•ด์ค€๋‹ค. URL์€ {jenkins URL}/jenkins/job/{job name}/buildWithParameters์‹์œผ๋กœ ์„ค์ •ํ•ด์ฃผ๊ณ  Content Type ์€ application/x-www-form-urlencoded์œผ๋กœ ์„ ํƒํ•œ๋‹ค. ์–ธ์ œ Webhook์„ ํŠธ๋ฆฌ๊ฑฐ๋ง ์‹œํ‚ฌ๊บผ๋ƒ๋Š” ์˜ต์…˜์—์„œ๋Š” ์›ํ•˜๋Š” ์„ค์ •์— ๋งž์ถ”๋ฉด ๋˜๊ฒ ์ง€๋งŒ ๋‚˜๋Š” pullRequest๊ฐ€ ๋“ฑ๋ก ๋ ๋•Œ๋งŒ ๋ฏธ๋ฆฌ ๋งŒ๋“ค์–ด ๋†“์€ Jenkins Job์„ ์‹คํ–‰์‹œํ‚ฌ ๊ณ„ํš์ด์˜€์œผ๋‹ˆ Let me select individual events.์„ ์„ค์ •ํ•˜๊ณ  Pull Request์— ์ฒดํฌ๋ฅผ ํ•ด์ค€๋‹ค. ์•„๋ž˜ ๊ทธ๋ฆผ์ฒ˜๋Ÿผ ๋ง์ด๋‹ค.

/images/github-web-hook-jenkins-job-excute/oss_webhook.png
ํ•ด๋‹น Repositroy > Settings > Hooks

์ด๋ ‡๊ฒŒ ๋“ฑ๋กํ•˜๊ณ  ๋‹ค์‹œ ๋“ค์–ด๊ฐ€์„œ ๋งจ ์•„๋žซ ๋ถ€๋ถ„Recent Deliveries์„ ๋ณด๋ฉด ping test ๊ฐ€ ์ด๋ฃจ์–ด์ ธ ์ •์ƒ์ ์œผ๋กœ ์‘๋‹ต์„ ๋ฐ›์€๊ฒƒ์„ ํ™•์ธํ• ์ˆ˜๊ฐ€ ์žˆ๋‹ค.

/images/github-web-hook-jenkins-job-excute/oss_webhook_result.png
Webhook ๋“ฑ๋ก ๊ฒฐ๊ณผ

์ด๋ ‡๊ฒŒ ์„ค์ •์„ ๋‹ค ํ•œ ๋’ค PullRequest๋ฅผ ๋ฐœ์ƒ์‹œํ‚ค๋ฉด Jenkins ํ•ด๋‹น Job์—์„œ๋Š” ํŒŒ๋ผ๋ฏธํ„ฐ๋ฅผ ๋ฐ›์œผ๋ฉฐ ์‹คํ–‰์ด ๋œ๊ฒƒ์„ ํ™•์ธํ• ์ˆ˜๊ฐ€ ์žˆ๋‹ค.

/images/github-web-hook-jenkins-job-excute/jenkins_webhook_result.png
Jenkins Job ์‹คํ–‰ ๊ฒฐ๊ณผ

๋~

Github๊ณผ Jenkins ์—ฐ๋™ํ•˜๊ธฐ

Jenkins์—์„œ Github์˜ ์†Œ์Šค๋ฅผ ๊ฐ€์ ธ์™€์„œ ๋นŒ๋“œ๋ฅผ ํ•˜๋Š” ๋“ฑ Github๊ณผ Jenkins์™€ ์—ฐ๋™์„ ์‹œ์ผœ์ค˜์•ผํ•˜๋Š” ์ƒํ™ฉ์—์„œ, ๋ณ„๋„์˜ ์„ ํ–‰ ์ž‘์—…์ด ํ•„์š”ํ•˜๋‹ค. ๋‹ค๋ฅธ ์—ฌ๋Ÿฌ ๋ฐฉ๋ฒ•์ด ์žˆ์„์ˆ˜ ์žˆ๋Š”๋ฐ ์—ฌ๊ธฐ์„œ๋Š” SSH๋กœ ์—ฐ๋™ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ์•„๋ณด๊ณ ์ž ํ•œ๋‹ค.

์šฐ์„  Jenkins๊ฐ€ ์„ค์น˜๋˜์–ด์žˆ๋Š” ์„œ๋ฒ„์—์„œ ์ธ์ฆํ‚ค๋ฅผ ์ƒ์„ฑํ•˜์ž.

$ ssh-keygen -t rsa -f id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
SHA256:~~~~~ ~~~@~~~~~
The key's randomart image is:
+---[RSA 2048]----+
|     o*+**=*=**+ |
|     o B=o+o++o  |
|     E+.o+ + oo .|
|      oo. * o ...|
|     .+ S  =   o |
|     . +    o .  |
|      . .  .     |
|       .         |
|                 |
+----[SHA256]-----+

$ ls
id_rsa  id_rsa.pub

๊ฐœ์ธํ‚ค(id_rsa)๋Š” ์  ํ‚จ์Šค์— ์„ค์ •ํ•ด์ค€๋‹ค. (์ฒ˜์Œ๋ถ€ํ„ฐ ๋๊นŒ์ง€ ๋ณต์‚ฌ, ์ฒซ์ค„ ๋งˆ์ง€๋ง‰์ค„ ๋นผ๋ฉด ์•ˆ๋œ๋‹ค… )

/images/github-with-jenkins/ssh_jenkins.png
์  ํ‚จ์Šค์— SSH ๊ฐœ์ธํ‚ค ์„ค์ •

๊ทธ ๋‹ค์Œ ๊ณต๊ฐœํ‚ค(id_rsa.pub)๋Š” Github์— ์„ค์ •์„ ํ•ด์ค€๋‹ค.

/images/github-with-jenkins/ssh_github.png
Github์— SSH ๊ณต๊ฐœํ‚ค ์„ค์ •

์ด๋ ‡๊ฒŒ ํ•œ๋’ค Jenkins ์—์„œ ์ž„์˜๋กœ job์„ ์ƒ์„ฑํ•˜๊ณ  job ์„ค์ • > ์†Œ์Šค์ฝ”๋“œ ๊ด€๋ฆฌ ์—์„œ git ๋ถ€๋ถ„์— ์•„๋ž˜์ฒ˜๋Ÿผ ํ…Œ์ŠคํŠธ๋ฅผ ํ•ด์„œ ์ •์ƒ์ ์œผ๋กœ ์—ฐ๋™์ด ๋œ๊ฒƒ์„ ํ™•์ธํ•œ๋‹ค. Credentials ๊ฐ’์„ ์œ„์—์„œ ์„ค์ •ํ•œ ๊ฐœ์ธํ‚ค๋กœ ์„ค์ •ํ•˜๊ณ , repo ์ฃผ์†Œ๋ฅผ SSH์šฉ์œผ๋กœ ์ ์—ˆ์„๋•Œ ์—๋Ÿฌ๊ฐ€ ์•ˆ๋‚˜์˜ค๋ฉด ์„ฑ๊ณตํ•œ๊ฒƒ์ด๋‹ค.

/images/github-with-jenkins/ssh_complete.png
์ •์ƒ ์—ฐ๊ฒฐ๋˜๋ฉด Jenkins ์˜ค๋ฅ˜๋„ ์—†๊ณ , github SSH ํ‚ค์— ๋…น์ƒ‰๋ถˆ์ด ๋“ค์–ด์˜จ๋‹ค.

๋~

linux(centOS)์—์„œ selenium ์„ค์ •ํ•˜๊ธฐ (feat. python)

ํ…Œ์ŠคํŠธ ์ฝ”๋“œ๋กœ ์•ˆ๋˜๋Š” ์‹ค์ œ ๋ธŒ๋ผ์šฐ์ €๋‹จ ์‚ฌ์šฉ์„ฑ ํ…Œ์ŠคํŠธ๋ฅผ ํ•˜๊ณ ์‹ถ์€ ๊ฒฝ์šฐ๊ฐ€ ์žˆ๋‹ค. ์ด๋ฅผํ…Œ๋ฉด ํ™”๋ฉด์ด ๋œจ๊ณ , ์–ด๋–ค ๋ฒ„ํŠผ์„ ๋ˆ„๋ฅด๋ฉด, ์–ด๋–ค ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์™€์•ผ ํ•˜๋Š” ์ผ๋ จ์˜ Regression Test. ์ด๋•Œ ํ™œ์šฉํ• ์ˆ˜ ์žˆ๋Š”๊ฒŒ ๋‹ค์–‘ํ•œ ๋„๊ตฌ๊ฐ€ ์žˆ์ง€๋งŒ ์ด๋ฒˆ์—” selenium ์— ๋Œ€ํ•ด์„œ ์•Œ์•„๋ณด๊ณ ์ž ํ•œ๋‹ค.

์ฒ˜์Œ๋ถ€ํ„ฐ ์‚ฌ์‹ค web application ํ…Œ์ŠคํŠธ๋ฅผ ํ•˜๋ ค๊ณ  selenium ๋ฅผ ์•Œ์•„๋ณด๊ฒŒ ๋œ๊ฑด ์•„๋‹ˆ๊ณ , ๋‚ด๊ฐ€ ์ฐธ์—ฌํ•˜๊ณ  ์žˆ๋Š” ํŠน์ • ๋ฐด๋“œ(๋„ค์ด๋ฒ„ BAND)์—์„œ ์ผ์ฃผ์ผ์— ํ•œ๋ฒˆ์”ฉ ๋™์ผํ•œ ํ˜•ํƒœ์˜ ๊ธ€์„ ์˜ฌ๋ฆฌ๊ณ  ์žˆ๋Š”๋ฐ (์ผ์ข…์˜ ํ•œ์ฃผ ์ถœ์„์ฒดํฌ ๊ฐ™์€…) ์ด๋ฅผ ์ž๋™ํ™” ํ•ด๋ณผ์ˆœ ์—†์„๊นŒ ํ•˜๋ฉฐ ๋ฐด๋“œ API๋ฅผ ์ฐพ์•„๋ณด๋‹ค selenium ๋ผ๋Š”๊ฒƒ์„ ์•Œ๊ฒŒ๋˜์—ˆ๊ณ , ๋งคํฌ๋กœ์ฒ˜๋Ÿผ ์–ด๋–ค๋ฒ„ํŠผ ๋ˆ„๋ฅด๊ณ  ๊ทธ๋‹ค์Œ ์–ด๋–ค๋ฒ„ํŠผ ๋ˆ„๋ฅด๊ณ  ํ•˜๋Š” ์ผ๋ จ์˜ ๊ณผ์ •์„ ์ฝ”๋“œ๋กœ ๊ตฌ์„ฑํ• ์ˆ˜ ์žˆ๋‹ค๋Š” ์ ์— ๊ฐ๋™์„ ๋ฐ›์•„(?) + ๋ณ„๋„์˜ API๋ฅผ ๋ฐœ๊ธ‰๋ฐ›์ง€ ์•Š์•„๋„ ๋˜์–ด ์‚ฌ์šฉํ•˜๊ฒŒ ๋˜์—ˆ๋‹ค. (๋ฌผ๋ก  UI๊ฐ€ ๋ฐ”๋€Œ๋ฉด ๊ณจ์น˜์•„ํ”„๊ฒ ์ง€๋งŒ…)

์—ฌ๊ธฐ์„œ๋Š” selenium ์ด ๋ฌด์—‡์ธ์ง€์— ๋Œ€ํ•œ ์„ค๋ช…์€ ํ•˜์ง€ ์•Š๋Š”๋‹ค. (์ธํ„ฐ๋„ท์— ๋‚˜๋ณด๋‹ค ์ •๋ฆฌ ์ž˜๋œ๊ธ€์ด ๋งŽ์œผ๋‹ˆ…) ๋‹จ, linux ํ™˜๊ฒฝ์—์„œ ์…‹ํŒ…ํ•˜๋Š” ์ •๋ณด๊ฐ€ ๋„ˆ๋ฌด ์—†๊ณ  ๋ช‡์ผ๋™์•ˆ ์‚ฝ์งˆ์„ ํ•œ๊ฒŒ ์•„์‰ฌ์›Œ์„œ ๊ทธ ๊ณผ์ •์„ ํฌ์ŠคํŒ… ํ•ด๋ณธ๋‹ค. (๋‚˜๊ฐ™์€ ๋ถ„์ด ์ด ๊ธ€์„ ๋ณด๊ณ  ๋„์›€์ด ๋˜์‹ค๊บผ๋ผ๋Š” ๊ธฐ๋Œ€๋ฅผ ๊ฐ–์œผ๋ฉฐ…)

โ€ป ์ฃผ์˜ : ๋ณธ ํฌ์ŠคํŒ…์€ ๋ฐด๋“œ ์„œ๋น„์Šค์— ๊ธ€์„ ์˜ฌ๋ฆด์ˆ˜ ์žˆ๋Š” ๋น„ ์ •์ƒ์ ์ธ ๋ฐฉ๋ฒ•์˜ ๊ณต์œ ๊ฐ€ ์•„๋‹Œ, selenium์— ๋Œ€ํ•œ ์‚ฌ์šฉ ํ›„๊ธฐ(?)์— ๋Œ€ํ•œ ๊ธ€์ž…๋‹ˆ๋‹ค. (์ฐธ๊ณ ๋กœ ๋ง‰ํ˜”์–ด์š” -ใ…-)

์„ค์ •ํ•˜๊ธฐ

์„œ๋ฒ„ ํ™˜๊ฒฝ์€ CentOS 7.4 64Bit + Python 3.6.3 + jdk 8 ์ด๋‹ค. ์šฐ์„  selenium ์„ ์„ค์น˜ํ•ด์ค€๋‹ค.

$ sudo python3.6 -m pip install selenium

๊ทธ ๋‹ค์Œ CentOS์—์„œ ํฌ๋กฌ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์„ค์น˜ํ•˜๊ธฐ ์œ„ํ•˜์—ฌ yum ์ €์žฅ์†Œ๋ฅผ ์ถ”๊ฐ€ํ•œ๋‹ค. (๊ผญ ํฌ๋กฌ์ด ์•„๋‹ˆ๋”๋ผ๋„ ํŒŒ์ด์–ดํญ์Šค๋‚˜ ์ง€๊ธˆ์€ ์ง€์›์ด ๋Š๊ธด ํŒฌํ…€JS ๊ฐ™์€๊ฒƒ์œผ๋กœ ํ™œ์šฉํ• ์ˆ˜๋„ ์žˆ์œผ๋‚˜ ๋‹ค๋ฅธ๊ฒƒ๋“ค๋„ ํ•ด๋ดค๋Š”๋ฐ ์ž๊พธ ์„ค์ •์—์„œ ๊ฑธ๋ ค์„œ ํฌ๋กฌ์— ๋Œ€ํ•œ ๋‚ด์šฉ์„ ํฌ์ŠคํŒ… ํ•œ๋‹ค.)

$ sudo vi /etc/yum.repos.d/google-chrome.repo
[google-chrome]
name=google-chrome
baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub

๊ทธ๋ฆฌ๊ณ ๋Š” yum ์œผ๋กœ ํฌ๋กฌ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์„ค์น˜ํ•œ๋‹ค. (๋‚ด๊ฐ€ ์„ค์น˜ํ–ˆ์„๋•Œ์˜ ๋ฒ„์ „์€ google-chrome-stable.x86_64 0:64.0.3282.119-1)

$ yum install google-chrome-stable

ํฌ๋กฌ๋“œ๋ผ์ด๋ฒ„๋ฅผ ์„ค์น˜ํ•ด์•ผํ•œ๋‹ค. ๋‹ค์Œ url์—์„œ ๋ฐ›์„์ˆ˜ ์žˆ๋Š”๋ฐ https://sites.google.com/a/chromium.org/chromedriver/downloads ๋‚˜๋Š” 2.35 linux64 ๋ฒ„์ „์„ ๋ฐ›์•˜๋‹ค. ๋‹ค์šด๋ฐ›๊ณ  unzip ํ•˜๋ฉด ๋”ฑํ•˜๋‚˜ ํŒŒ์ผ์ด ์žˆ๋Š”๋ฐ ๋‚˜์ค‘์— selenium ์„ ์‚ฌ์šฉํ• ๋•Œ ์ด์šฉ๋˜๋‹ˆ path๋ฅผ ์•Œ์•„๋‘์ž.

๊ทธ๋‹ค์Œ ํŒŒ์ด์ฌ ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•œ๋‹ค. ๋‚ด๊ฐ€ ์ง  ํŒŒ์ด์ฌ ์ฝ”๋“œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ˆœ์„œ๋กœ ์‹คํ–‰์ด ๋œ๋‹ค.

  • ๋ฐด๋“œ ์ ‘์† ( https://band.us/home )
  • ๋กœ๊ทธ์ธ
  • ํŠน์ • ๋ฐด๋“œ ์„ ํƒ
  • ๊ธ€์“ฐ๊ธฐ ๋ฒ„ํŠผ ๋ˆ„๋ฅด๊ณ  ์–‘์‹์— ๋งž์ถฐ ๊ธ€ ์ž‘์„ฑ
  • ๊ธ€ ๋“ฑ๋ก

ํŒŒ์ด์ฌ ์ฝ”๋“œ๋Š” ์•„๋ž˜์ฒ˜๋Ÿผ ์ž‘์„ฑํ•˜์˜€๋‹ค. (์ค‘์š”๋ถ€๋ถ„๋งŒ.. ๊ทธ ์•„๋ž˜๋Š” ์ž์œ )

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# ์ดˆ๊ธฐํ™” --------------------------------------------
chrome_options = Options()
chrome_options.add_argument("--headless")

driver  = webdriver.Chrome(executable_path='home/~~~/chromedriver', chrome_options=chrome_options)
driver.implicitly_wait(3)
driver.get('https://band.us/home')

# ๋กœ๊ทธ์ธ --------------------------------------------
driver.find_element_by_class_name('_loginLink').click()
...์ƒ๋žต

๊ทธ๋ฆฌ๊ณ  ์‹คํ–‰์„ ํ•ด๋ณด๋ฉด ์ž‘๋™์ด ์ž˜~ ๋œ๋‹ค.

/images/linux-selenium/result.png
selenium + python ์œผ๋กœ ์ž๋™์ž‘์„ฑ๋œ ๋ฐด๋“œ ๊ธ€

๋งˆ์น˜๋ฉฐ

selenium ์— ๋Œ€ํ•ด ์ฐพ์•„๋ณด๋ฉด ๊ฑฐ์˜ ์œˆ๋„์šฐ ํ™˜๊ฒฝ์—์„œ ๋Œ์•„๊ฐ€๋Š”๊ฒƒ๋“ค์— ๋Œ€ํ•œ ํฌ์ŠคํŒ…์ด ๋งŽ์•˜๋‹ค. ๋‚œ ๋ฆฌ๋ˆ…์Šค ํ™˜๊ฒฝ์—์„œ ์Šค์ผ€์ฅด๋Ÿฌ(์  ํ‚จ์Šค ๊ฐ™์€)๋ฅผ ํ†ตํ•ด ์ž๋™์œผ๋กœ ํ™”๋ฉด์—†์ด ์ž‘๋™์‹œํ‚ค๊ณ  ์‹ถ์—ˆ๋Š”๋ฐ ์•„๋ฌด๋ฆฌ ์ฐพ์•„๋ด๋„ + ์‚ฝ์งˆํ•ด๋„ ์ž˜ ์•ˆ๋˜์—ˆ๋‹ค. ๊ฒฐ๊ตญ ์‚ฌ๋‚ด์—๋„ ๋‚˜๊ฐ™์€ ์‚ฝ์งˆ์„ ํ•˜์‹  ๋ถ„์„ ์ฐพ๊ณ  ๋ฌป๊ณ  ๋ฌผ์–ด ํฌ๋กฌ๋“œ๋ผ์ด๋ฒ„๋งŒ ์žˆ์–ด์•ผ ํ•˜๋Š”๊ฒƒ์ด ์•„๋‹ˆ๋ผ ํฌ๋กฌ์•ฑ๋˜ํ•œ ์žˆ์–ด์•ผ ๋™์ž‘์„ ํ•œ๋‹ค๋Š”๊ฒƒ์„ ์•Œ๊ฒŒ ๋˜์—ˆ๋‹ค. ์—ญ์‹œ, ๋‚ด๊ฐ€ ํ•œ ์‚ฝ์งˆ์€ ๋ˆ„๊ตฐ๊ฐ€ ์ด๋ฏธ ํ•œ ์‚ฝ์งˆ์ด๋ผ๋Š”๊ฑธ ๋‹ค์‹œํ•œ๋ฒˆ ๊นจ๋‹ณ์€ ์ข‹์€(?) ์‹œ๊ฐ„์ด์˜€๋‹ค.

์ด๊ฑธ๋กœ ๋‚˜์ค‘์— ๋‚ด๊ฐ€ ๋งก๊ณ ์žˆ๋Š” ์„œ๋น„์Šค์— ๋Œ€ํ•œ ์›น ์ž๋™ ํ…Œ์ŠคํŠธ ํˆด๋„ ๋งŒ๋“ค์–ด ๋ณผ ์ƒ๊ฐ์ด๋‹ค.

์•„ํŒŒ์น˜ ์—‘์„ธ์Šค ๋กœ๊ทธ๋ฅผ ์—˜๋ผ์Šคํ‹ฑ์„œ์น˜์— ์ธ๋ฑ์‹ฑ ํ•ด๋ณด์ž.

apache access log ๋ฅผ ๋ถ„์„ํ•˜๊ณ  ์‹ถ์€ ์ƒํ™ฉ์ด ์ƒ๊ฒผ๋‹ค. ์•„๋‹ˆ ๊ทธ๋ณด๋‹ค apache access์— ๋Œ€ํ•ด์„œ ์‹ค์‹œ๊ฐ„์œผ๋กœ ๋ณด๊ณ ์‹ถ์—ˆ๊ณ , log๋ฅผ ๊ฒ€์ƒ‰ & ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€๊ณตํ•˜์—ฌ ์œ ์˜๋ฏธํ•œ ๋ถ„์„๊ฒฐ๊ณผ๋ฅผ ๋งŒ๋“ค์–ด ๋ณด๊ณ  ์‹ถ์—ˆ๋‹ค. ๊ทธ์— ์ƒ๊ฐํ•œ๊ฒƒ์ด (์—ญ์‹œ) ElasticStack.

์ฒ˜์Œ์— ์ƒ๊ฐํ•œ ๋ฐฉ์•ˆ์€ ์•„๋ž˜ ๊ทธ๋ฆผ์ฒ˜๋Ÿผ ๋‹จ์ˆœํ–ˆ๋‹ค.

/images/apache-access-log-to-es/model_1.png
์ฒ˜์Œ ์ƒ๊ฐํ•œ ๋‹จ์ˆœํ•œ ๊ตฌ์กฐ

ํ•˜์ง€๋งŒ, ๋‚ด ๋‹จ์ˆœํ•œ(?) ์˜ˆ์ƒ์€ ์—ญ์‹œ ๋น—๋‚˜๊ฐ”๊ณ  logstash์—์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์—๋Ÿฌ๋ฅผ ๋‚ด๋ฑ‰์—ˆ๋‹ค.

retrying individual bulk actions that failed or were rejected by the previous bulk request

request๊ฐ€ ๋งŽ์•„์ง์— ๋”ฐ๋ผ elasticsearch๊ฐ€ ๋ฒ„๋ฒ…๊ฑฐ๋ฆฌ๋”๋‹ˆ logstash์—์„œ ๋Œ€๋Ÿ‰์ž‘์—…์€ ๊ฑฐ๋ถ€ํ•˜๊ฒ ๋‹ค๋ฉฐ ์ธ๋ฑ์‹ฑ์„ ๋ฉˆ์ท„๋‹ค. ๊ณ ๋ฏผ๊ณ ๋ฏผํ•˜๋‹ค elasticsearch์— ์ธ๋ฑ์‹ฑํ• ๋•Œ ๋ถ€ํ•˜๊ฐ€ ๋งŽ์ด ๊ฑธ๋ฆฌ๋Š” ์ƒํ™ฉ์—์„œ ์ค‘๊ฐ„์— ๋ฒ„ํผ๋ฅผ ๋‘” ๊ฒฝํ—˜์ด ์žˆ์–ด์„œ facebook๊ทธ๋ฃน์— ๋ฌธ์˜๋ฅผ ํ•ด๋ดค๋‹ค. https://www.facebook.com/groups/elasticsearch.kr/?multi_permalinks=1566735266745641 ์—ญ์‹œ ๋‚˜๋ณด๋‹ค ํ•œ์ฐธ์„ ์•ž์„œ๊ฐ€์‹œ๋Š” ๋ถ„๋“ค์€ ์ด๋ฏธ ์—๋Ÿฌ๊ฐ€ ๋ญ”์ง€ ์•Œ๊ณ  ์žˆ์œผ์…จ๊ณ , ์ค‘๊ฐ„์— ๋ฒ„ํผ๋ฅผ ๋‘๊ณ  ํ•˜๋‹ˆ ์ž˜๋œ๋‹ค๋Š” ์˜๊ฒฌ์ด ์žˆ์–ด ๋‚˜๋„ ๋”ฐ๋ผํ•ด๋ดค๋‹ค. ๋ฌผ๋ก  ๋‹ต๋ณ€์ค‘์— ๋‚˜์˜จ redis๊ฐ€ ์•„๋‹Œ ๊ธฐ์กด์—๋„ ๋น„์Šทํ•œ ๊ตฌ์กฐ์—์„œ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋˜ kafka๋ฅผ ์ ์šฉ. ์•„, ๊ทธ์ „์— ํ˜„์žฌ๊ตฌ์„ฑ์€ Elasticsearch ๋…ธ๋“œ๊ฐ€ ์ด 3๋Œ€๋กœ ํด๋Ÿฌ์Šคํ„ฐ ๊ตฌ์กฐ๋กœ ๋˜์–ด์žˆ๋Š”๋ฐ ๋…ธ๋“œ๋ฅผ ์ถ”๊ฐ€๋กœ ๋Š˜๋ฆฌ๋ฉฐ ์Šค์ผ€์ผ ์•„์›ƒ์„ ํ•ด๋ณด๊ธฐ์ „์— ํ• ์ˆ˜์žˆ๋Š” ๋งˆ์ง€๋ง‰ ๋ฐฉ๋ฒ•์ด๋‹ค ์ƒ๊ฐํ•˜๊ณ  ์ค‘๊ฐ„์— kafka๋ฅผ ๋‘ฌ์„œ ๋ถ€ํ•˜๋ฅผ ์ค„์—ฌ๋ณด๊ณ  ์‹ถ์—ˆ๋‹ค. (์–ธ์ œ๋ถ€ํ„ด๊ฐ€ ๋งˆ์น˜ ์—ฌ๋Ÿฌ๊ฐœ์˜ ํ†ฑ๋‹ˆ๋ฐ”ํ€ด๊ฐ€ ๋งž๋ฌผ๋ ค ๋Œ์•„๊ฐ€๋Š”๋“ฏํ•œ ์‹œ์Šคํ…œ ์„ค๊ณ„๋ฅผ ํ•˜๋Š”๊ฒŒ ์žฌ๋ฐŒ์—ˆ๋‹ค.) ์•„๋ž˜ ๊ทธ๋ฆผ์ฒ˜๋Ÿผ ๋ง์ด๋‹ค.

/images/apache-access-log-to-es/model_2.png
๊ทธ๋‚˜๋งˆ ์ข€๋” ์ƒ๊ฐํ•œ ๊ตฌ์กฐ

๊ทธ๋žฌ๋”๋‹ˆ ๊ฑฐ์ง“๋ง ์ฒ˜๋Ÿผ ์—๋Ÿฌํ•˜๋‚˜ ์—†์ด ์ž˜ ์ธ๋ฑ์‹ฑ์ด ๋ ์ˆ˜ ์žˆ์—ˆ๋‹ค. logstash๊ฐ€ ์–‘์ชฝ์— ์žˆ๋Š”๊ฒŒ ์•ฝ๊ฐ„ ๊ฑธ๋ฆฌ๊ธด ํ•˜์ง€๋งŒ, ์ฒ˜์Œ์— ์ƒ๊ฐํ•œ ๊ตฌ์กฐ๋ณด๋‹ค๋Š” ์—๋Ÿฌ๊ฐ€ ์•ˆ๋‚˜๋‹ˆ ๋‹คํ–‰์ด๋ผ ์ƒ๊ฐํ•œ๋‹ค.

์ด ๊ตฌ์กฐ๋ฅผ ์ ์šฉํ•˜๋ฉด์„œ ์–ป์€ Insight๊ฐ€ ์žˆ๊ธฐ์—, ๊ฐ ํ•ญ๋ชฉ๋ณ„๋กœ ์ ์–ด ๋ณด๊ณ ์ž ํ•œ๋‹ค. ( ์ด๊ฒƒ๋งŒ ์ ์–ด๋†“๊ธฐ์—” ๋„ˆ๋ฌด ์—†์–ด๋ณด์—ฌ์„œ.. )

access log ๋ฅผ ์–ด๋–ป๊ฒŒ ๋ถ„์„ํ•˜์—ฌ ์ธ๋ฑ์‹ฑ ํ• ๊ฒƒ์ธ๊ฐ€?

apache 2.x๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ๋ณ„๋„์˜ ๋กœ๊ทธ ํฌ๋งท์„ ์ •ํ•˜์ง€ ์•Š์œผ๋ฉด ์•„๋ž˜์™€ ๊ฐ™์€ access log๊ฐ€ ์ฐํžŒ๋‹ค. 123.1.1.1 - - [25/Jan/2018:21:55:35 +0900] "GET /api/test?param=12341234 HTTP/1.1" 200 48 1144 "http://www.naver.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 NAVER(inapp; blog; 100; 4.0.44)" ๊ทธ๋Ÿผ ์ด ๋กœ๊ทธ๋ฅผ ์•„๋ฌด ํฌ๋งทํŒ… ์—†์ด ๋กœ๊น…์„ ํ•˜๋ฉด ๊ทธ๋ƒฅ ํ•œ์ค„์˜ ํ…์ŠคํŠธ๊ฐ€ ์ธ๋ฑ์‹ฑ์ด ๋œ๋‹ค. ํ•˜์ง€๋งŒ ์ด๋ ‡๊ฒŒ ๋˜๋ฉด elasticsearch ๋ฐ์ดํ„ฐ๋ฅผ ๋‹ค์‹œ ์žฌ๊ฐ€๊ณตํ•˜๊ฑฐ๋‚˜ ๋ณ„๋„์˜ ์ž‘์—…์ด ํ•„์š”ํ• ์ˆ˜๋„ ์žˆ์œผ๋‹ˆ ์ค‘๊ฐ„์— ์žˆ๋Š” logstash์—๊ฒŒ ์ผ์„ ์‹œ์ผœ ์ข€๋” nice ํ•œ ๋ฐฉ๋ฒ•์œผ๋กœ ์ธ๋ฑ์‹ฑ์„ ํ•ด๋ณด์ž. ๋ฐ”๋กœ logstash ์˜ filter ๊ธฐ๋Šฅ์ด๋‹ค. ๊ทธ์ค‘ Grok filter ๋ผ๋Š”๊ฒŒ ์žˆ๋Š”๋ฐ ํŒจํ„ด์„ ์ ์šฉํ•˜์—ฌ row data ๋ฅผ ํ•„ํ„ฐ๋งํ•˜๋Š” ๊ธฐ๋Šฅ์ด๋‹ค. ์กฐ๊ธˆ ์ฐพ์•„๋ณด๋‹ˆ ๋„ˆ๋ฌด ๊ณ ๋ง™๊ฒŒ๋„ ์•„ํŒŒ์น˜ ํ•„ํ„ฐ ์˜ˆ์ œ๊ฐ€ ์žˆ์–ด ์ˆ˜์ •ํ•˜์—ฌ ์ ์šฉํ• ์ˆ˜ ์žˆ์—ˆ๋‹ค. http://grokconstructor.appspot.com/do/match?example=2 ๊ทธ๋ž˜์„œ ์ ์šฉํ•œ ํ•„ํ„ฐ์„ค์ •์€ ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค.

filter {
        grok {
                match => {
                        message => "%{IP:clientIp} (?:-|) (?:-|) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:httpMethod} %{URIPATH:uri}%{GREEDYDATA}(?: HTTP/%{NUMBER})?|-)\" %{NUMBER:responseCode} (?:-|%{NUMBER})"
                }
        }
}

์ด๋ ‡๊ฒŒ ํ•˜๊ณ  elasticsearch ์— ์ธ๋ฑ์‹ฑ์„ ํ•˜๋ฉด ํ‚ค๋ฐ”๋‚˜์—์„œ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋ณผ์ˆ˜ ์žˆ๋‹ค.

/images/apache-access-log-to-es/access_log_kibana.png
ํ‚ค๋ฐ”๋‚˜์— ๋‚ด๊ฐ€ ์›ํ•˜๋Š” ๊ตฌ์กฐ๋Œ€๋กœ ์ด์˜๊ฒŒ ๋“ค์–ด๊ฐ€ ์žˆ๋Š” access log

๊ฐ ํ•„๋“œ๊ฐ€ ์•„๋‹Œ ํ•œ์ค„๋กœ ์ธ๋ฑ์‹ฑ์ด ๋˜์–ด๋ฒ„๋ฆฐ๋‹ค.

Elasticsearch ์— ์ธ๋ฑ์‹ฑ์ด ๋˜๊ธด ํ•˜๋Š”๋ฐ ๋กœ๊ทธ ํ•œ์ค„์ด ํ†ต์งธ๋กœ ๋“ค์–ด๊ฐ€ ๋ฒ„๋ฆฐ๋‹ค. message๋ผ๋Š” ์ด๋ฆ„์œผ๋กœ… ์•Œ๊ณ ๋ณด๋‹ˆ ํ˜„์žฌ ๊ตฌ์กฐ๋Š” logstash๊ฐ€ kafka ์•ž ๋’ค์— ์žˆ๋‹ค๋ณด๋‹ˆ producer logstash ์™€ consumer logstash ์˜ codec์ด ๋งž์•„์•ผ ์ œ๋Œ€๋กœ ์ธ๋ฑ์‹ฑ์ด ๋ ์ˆ˜ ์žˆ์—ˆ๋‹ค. ๋จผ์ € access log์—์„œ kafka ๋กœ produce ํ•˜๋Š” logstash ์—์„œ๋Š” output ํ• ๋•Œ codec ์„ ๋งž์ถฐ์ฃผ๊ณ 

output {
        kafka {
                bootstrap_servers => "123.1.2.3:9092,123.1.2.4:9092"
                topic_id => "apache-log"
                codec => json{}
        }
}

kafka ์—์„œ consume ํ•˜๋Š” logstash ์—์„œ๋Š” input ์—์„œ codec ์„ ๋งž์ถฐ์ค€๋‹ค.

input {
        kafka {
                bootstrap_servers => "123.1.2.3:9092,123.1.2.4:9092"
                topic_id => "apache-log"
                codec => json{}
        }
}

๊ทธ๋ ‡๊ฒŒ ๋˜๋ฉด codec์ด ๋งž์•„ ๊ฐ ํ•„๋“œ๋กœ ์ด์˜๊ฒŒ ์ธ๋ฑ์‹ฑ์„ ํ• ์ˆ˜ ์žˆ๊ฒŒ ๋˜์—ˆ๋‹ค.

ํŒŒ์ด์ฌ ๋ฒ„์ „ ์—…๊ทธ๋ ˆ์ด๋“œ (2.6 > 3.6)

ํŒŒ์ด์ฌ 2.x ์—์„œ๋Š” depreate ๋œ ๋ชจ๋“ˆ๋„ ๋งŽ๊ณ  3.x์—์„œ๋งŒ ์ง€์›๋˜๋Š” ๋ฒ„์ „๋“ค์ด ๋งŽ์•„์ง€๋ฉด์„œ ์‹ค์ปท ๊ฐœ๋ฐœ์„ ํ•ด๋„ ํŒŒ์ด์ฌ ๋ฒ„์ „๋•Œ๋ฌธ์— ๋‹ค์‹œ ์งœ์•ผํ•˜๋Š” ์ƒํ™ฉ์ด ์ƒ๊ธด๋‹ค. ํŒŒ์ด์ฌ ๋ฒ„์ „์—…์„ ํ•˜๊ณ ์‹ถ์–ด ๊ตฌ๊ธ€๋ง์„ ํ•ด๋ณด๋ฉด ์ด๋ ‡๋‹คํ•  ์ •๋ฆฌ๋œ ๋ฌธ์„œ๊ฐ€ ์ž˜ ์•ˆ๋‚˜์˜จ๋‹ค. (์˜์–ด๋กœ๋œ ํฌ์ŠคํŠธ๋Š” ๋งŽ์ด ์žˆ๊ธด ํ•˜๋‚˜, ํ•„์ž์˜ ํ™˜๊ฒฝ๊ณผ๋Š” ๋งž์ง€ ์•Š๋Š” …) ๊ทธ๋ž˜์„œ ์ด๊ฒƒ์ €๊ฒƒ ์‚ฝ์งˆ์„ ํ•œ ๊ฒฐ๊ณผ ํŒŒ์ด์ฌ ๋ฒ„์ „์„ ์˜ฌ๋ฆด์ˆ˜ ์žˆ์—ˆ๊ณ , ์ด๋ฅผ ํฌ์ŠคํŒ… ํ•ด๋ณด๊ณ ์ž ํ•œ๋‹ค.

๊ทธ๋Ÿฌ๋ณด๊ณ ๋‹ˆ 2018๋…„ ์ฒซ ํฌ์ŠคํŒ…์ด๋„ค… ์˜ฌํ•ด๋Š” ์ •๋ง ์ ์–ด๋„ ํ•œ๋‹ฌ์— 1~2๊ฐœ๋Š” ์˜ฌ๋ฆด์ˆ˜ ์žˆ๋Š” ๋‚ด๊ฐ€ ๋˜๊ธฐ๋ฅผ…

ํ™˜๊ฒฝ

  • CentOS 6.9
  • ๊ธฐ๋ณธ์œผ๋กœ python 2.6 ์ด ์„ค์น˜๋˜์–ด ์žˆ๋Š”๊ฒƒ์„ ํ™•์ธํ• ์ˆ˜ ์žˆ๋‹ค. (ํ™˜๊ฒฝ๋งˆ๋‹ค ๋‹ค๋ฅผ์ˆ˜ ์žˆ์Œ.)
$ python -V
Python 2.6

์„ค์น˜์ˆœ์„œ

  • ํ•„์š”ํ•œ ์œ ํ‹ธ๋ฆฌํ‹ฐ๋ฅผ ์„ค์น˜ํ•œ๋‹ค.
$ sudo yum update
$ sudo yum install yum-utils
$ sudo yum groupinstall development
  • yum ์ €์žฅ์†Œ์—์„œ๋Š” ์ตœ์‹  ํŒŒ์ด์ฌ ๋ฆด๋ฆฌ์ฆˆ๋ฅผ ์ œ๊ณตํ•˜์ง€ ์•Š์œผ๋ฏ€๋กœ RPM ํŒจํ‚ค๋ฅผ ์ œ๊ณตํ•˜๋Š” IUM ์ด๋ผ๋Š” ์ถ”๊ฐ€ ์ €์žฅ์†Œ๋ฅผ ์„ค์น˜
$ sudo yum install -y https://repo.ius.io/ius-release-el7.rpm
  • ํŒŒ์ด์ฌ 3.6 ๋ฒ„์ „์„ ์„ค์น˜
$ sudo yum install python36u
  • pip ๋“ฑ ํŒจํ‚ค์ง€ ๊ด€๋ จ ๋ชจ๋“ˆ๋„ ํ•จ๊ป˜ ์„ค์น˜
sudo yum install python36u-pip
sudo yum install python36u-devel
  • ์—ฌ๊ธฐ๊นŒ์ง€ ํ•˜๋ฉด ๊ธฐ์กด ํŒŒ์ด์ฌ 2.6๊ณผ ์ƒˆ๋กœ ์„ค์น˜๋œ ํŒŒ์ด์ฌ 3.6 ์ด ์„ค์น˜๋˜์–ด์žˆ๋‹ค.
$ ll /usr/bin/python*
-rwxr-xr-x 1 root root 9997450 Jan  2 16:02 python
lrwxrwxrwx 1 root root       6 Jan  1 06:02 python2 -> python
-rwxr-xr-x 1 root root    9032 Aug 19  2016 python2.6
-rwxr-xr-x 1 root root    1418 Aug 19  2016 python2.6-config
-rwxr-xr-x 2 root root    6808 Oct 12 08:19 python3.6
lrwxrwxrwx 1 root root      26 Jan  2 20:48 python3.6-config -> /usr/bin/python3.6m-config
-rwxr-xr-x 2 root root    6808 Oct 12 08:19 python3.6m
-rwxr-xr-x 1 root root     173 Oct 12 08:19 python3.6m-config
-rwxr-xr-x 1 root root    3339 Oct 12 08:16 python3.6m-x86_64-config
lrwxrwxrwx 1 root root      16 Apr 25  2017 python-config -> python2.6-config
  • ํ™˜๊ฒฝ๋ณ€์ˆ˜๋ฅผ ์„ค์ •ํ•ด์ค€๋‹ค.
$ sudo mv python python_backup
$ sudo ln -s python3.6 python
  • ํ™•์ธ
$ python -V
Python 3.6.3

pip ๋ฅผ ์ด์šฉํ•œ ๋ชจ๋“ˆ ์„ค์น˜

  • pip๋ž€ Python Package Index ์˜ ์•ฝ์ž๋กœ ๊ณต์‹ํ™ˆํŽ˜์ด์ง€๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค. ( https://pypi.python.org/pypi/pip )
  • ์„ค์น˜ํ•  ๋ชจ๋“ˆ์„ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์„ค์น˜ํ•ด์ฃผ๋ฉด ๋œ๋‹ค. ex : requests ๋ชจ๋“ˆ์ธ ๊ฒฝ์šฐ
$ sudo python3.6 -m pip install requests